[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:25:06 dist-test-slave-gjrz krb5kdc[30134](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:25:06 dist-test-slave-gjrz krb5kdc[30134](info): set up 2 sockets
Aug 11 22:25:06 dist-test-slave-gjrz krb5kdc[30134](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260811 22:25:07.419960 30120 mini_kdc.cc:121] Time spent starting KDC: real 1.465s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Aug 11 22:25:07 dist-test-slave-gjrz krb5kdc[30134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:25:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:25:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.106.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
I20260811 22:25:07.595531 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:38933
--webserver_interface=127.29.106.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:44007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:38933
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:25:08.273204 30162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:08.273651 30162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:08.273906 30162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:08.284689 30162 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:25:08.284864 30162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:08.284968 30162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:08.285081 30162 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:25:08.285159 30162 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:25:08.299628 30162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:44007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:38933
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:38933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.106.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:25:08.301760 30162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:08.303803 30162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:08.324613 30173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:08.324692 30175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:08.324760 30178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:08.326000 30162 server_base.cc:1034] running on GCE node
I20260811 22:25:08.327100 30162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:08.329526 30162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:08.330809 30162 hybrid_clock.cc:648] HybridClock initialized: now 1786487108330681 us; error 66 us; skew 500 ppm
I20260811 22:25:08.331429 30162 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:25:08 dist-test-slave-gjrz krb5kdc[30134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487108, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:08.336206 30162 init.cc:377] Logged in from keytab as kudu/127.29.106.62@KRBTEST.COM (short username kudu)
I20260811 22:25:08.339080 30162 webserver.cc:460] Webserver started at http://127.29.106.62:44567/ using document root <none> and password file <none>
I20260811 22:25:08.340355 30162 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:08.340498 30162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:08.340974 30162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:08.345821 30162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "a815aa29686a46acbfa692628a6c9d0b"
format_stamp: "Formatted at 2026-08-11 22:25:08 on dist-test-slave-gjrz"
server_key: "badb35a8724160ce5cf4dd60eca880aa"
server_key_iv: "7407c508b8e5e74ae76172682a70cd8b"
server_key_version: "encryptionkey@0"
I20260811 22:25:08.346590 30162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "a815aa29686a46acbfa692628a6c9d0b"
format_stamp: "Formatted at 2026-08-11 22:25:08 on dist-test-slave-gjrz"
server_key: "badb35a8724160ce5cf4dd60eca880aa"
server_key_iv: "7407c508b8e5e74ae76172682a70cd8b"
server_key_version: "encryptionkey@0"
I20260811 22:25:08.355672 30162 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.010s sys 0.001s
I20260811 22:25:08.362604 30194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:08.364779 30162 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260811 22:25:08.365051 30162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "a815aa29686a46acbfa692628a6c9d0b"
format_stamp: "Formatted at 2026-08-11 22:25:08 on dist-test-slave-gjrz"
server_key: "badb35a8724160ce5cf4dd60eca880aa"
server_key_iv: "7407c508b8e5e74ae76172682a70cd8b"
server_key_version: "encryptionkey@0"
I20260811 22:25:08.365391 30162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:08.384491 30162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:08.391191 30162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:08.391816 30162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:08.442550 30162 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.62:38933
I20260811 22:25:08.442584 30287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.62:38933 every 8 connection(s)
I20260811 22:25:08.445411 30162 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:25:08.446977 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 30162
I20260811 22:25:08.447436 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:25:08.450084 30120 external_mini_cluster.cc:1428] Setting key 90f11f82586b4ae476def74ac682aa80
I20260811 22:25:08.454756 30288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:08.475884 30288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a815aa29686a46acbfa692628a6c9d0b: Bootstrap starting.
I20260811 22:25:08.483517 30288 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a815aa29686a46acbfa692628a6c9d0b: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:08.485911 30288 log.cc:824] T 00000000000000000000000000000000 P a815aa29686a46acbfa692628a6c9d0b: Log is configured to *not* fsync() on all Append() calls
Aug 11 22:25:08 dist-test-slave-gjrz krb5kdc[30134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:25:08.490499 30288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a815aa29686a46acbfa692628a6c9d0b: No bootstrap required, opened a new log
I20260811 22:25:08.503743 30295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:08.456582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50782 (local address 127.29.106.62:38933)
0811 22:25:08.458680 (+ 2098us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:08.458711 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:08.459897 (+ 1186us) server_negotiation.cc:407] Connection header received
0811 22:25:08.473940 (+ 14043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:08.473990 (+ 50us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:08.474560 (+ 570us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:08.474998 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:08.479695 (+ 4697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:08.481649 (+ 1954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:08.483659 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:08.484488 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:08.484764 (+ 276us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:08.492169 (+ 7405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:08.492229 (+ 60us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:08.492267 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:08.492322 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:08.497072 (+ 4750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:08.497963 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:08.497970 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:08.497976 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:08.498278 (+ 302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:08.498848 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:08.498853 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:08.498855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:08.499760 (+ 905us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:08.500266 (+ 506us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:08.501355 (+ 1089us) server_negotiation.cc:299] Negotiation successful
0811 22:25:08.501801 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1645,"thread_start_us":912,"threads_started":1}
I20260811 22:25:08.504696 30288 raft_consensus.cc:348] T 00000000000000000000000000000000 P a815aa29686a46acbfa692628a6c9d0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a815aa29686a46acbfa692628a6c9d0b" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 38933 } }
I20260811 22:25:08.505334 30288 raft_consensus.cc:374] T 00000000000000000000000000000000 P a815aa29686a46acbfa692628a6c9d0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:25:08.505585 30288 raft_consensus.cc:729] T 00000000000000000000000000000000 P a815aa29686a46acbfa692628a6c9d0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a815aa29686a46acbfa692628a6c9d0b, State: Initialized, Role: FOLLOWER
I20260811 22:25:08.507180 30288 consensus_queue.cc:260] T 00000000000000000000000000000000 P a815aa29686a46acbfa692628a6c9d0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a815aa29686a46acbfa692628a6c9d0b" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 38933 } }
I20260811 22:25:08.507769 30288 raft_consensus.cc:388] T 00000000000000000000000000000000 P a815aa29686a46acbfa692628a6c9d0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:25:08.508198 30288 raft_consensus.cc:482] T 00000000000000000000000000000000 P a815aa29686a46acbfa692628a6c9d0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:25:08.508621 30288 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a815aa29686a46acbfa692628a6c9d0b [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:08.512048 30288 raft_consensus.cc:504] T 00000000000000000000000000000000 P a815aa29686a46acbfa692628a6c9d0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a815aa29686a46acbfa692628a6c9d0b" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 38933 } }
I20260811 22:25:08.513111 30288 leader_election.cc:302] T 00000000000000000000000000000000 P a815aa29686a46acbfa692628a6c9d0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a815aa29686a46acbfa692628a6c9d0b; no voters:
I20260811 22:25:08.514405 30288 leader_election.cc:288] T 00000000000000000000000000000000 P a815aa29686a46acbfa692628a6c9d0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:25:08.514675 30301 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a815aa29686a46acbfa692628a6c9d0b [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:25:08.517637 30301 raft_consensus.cc:686] T 00000000000000000000000000000000 P a815aa29686a46acbfa692628a6c9d0b [term 1 LEADER]: Becoming Leader. State: Replica: a815aa29686a46acbfa692628a6c9d0b, State: Running, Role: LEADER
I20260811 22:25:08.518755 30301 consensus_queue.cc:237] T 00000000000000000000000000000000 P a815aa29686a46acbfa692628a6c9d0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a815aa29686a46acbfa692628a6c9d0b" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 38933 } }
I20260811 22:25:08.520674 30288 sys_catalog.cc:564] T 00000000000000000000000000000000 P a815aa29686a46acbfa692628a6c9d0b [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:25:08.524989 30303 sys_catalog.cc:455] T 00000000000000000000000000000000 P a815aa29686a46acbfa692628a6c9d0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a815aa29686a46acbfa692628a6c9d0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a815aa29686a46acbfa692628a6c9d0b" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 38933 } } }
I20260811 22:25:08.525516 30303 sys_catalog.cc:458] T 00000000000000000000000000000000 P a815aa29686a46acbfa692628a6c9d0b [sys.catalog]: This master's current role is: LEADER
I20260811 22:25:08.526563 30302 sys_catalog.cc:455] T 00000000000000000000000000000000 P a815aa29686a46acbfa692628a6c9d0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a815aa29686a46acbfa692628a6c9d0b. Latest consensus state: current_term: 1 leader_uuid: "a815aa29686a46acbfa692628a6c9d0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a815aa29686a46acbfa692628a6c9d0b" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 38933 } } }
I20260811 22:25:08.527818 30302 sys_catalog.cc:458] T 00000000000000000000000000000000 P a815aa29686a46acbfa692628a6c9d0b [sys.catalog]: This master's current role is: LEADER
I20260811 22:25:08.528978 30314 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:25:08.540088 30314 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:25:08.557134 30314 catalog_manager.cc:1347] Generated new cluster ID: 9fbb2d9d8fae42f1b59c9708fa547db2
I20260811 22:25:08.557301 30314 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:25:08.566377 30314 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:25:08.567852 30314 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:25:08.578492 30314 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a815aa29686a46acbfa692628a6c9d0b: Generated new TSK 0
I20260811 22:25:08.579406 30314 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.106.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab.
Entry for principal kudu/127.29.106.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab.
I20260811 22:25:08.634745 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.1:0
--local_ip_for_outbound_sockets=127.29.106.1
--webserver_interface=127.29.106.1
--webserver_port=0
--tserver_master_addrs=127.29.106.62:38933
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab
--principal=kudu/127.29.106.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:44007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:25:09.276393 30335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:09.276791 30335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:09.277026 30335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:09.287997 30335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:09.288223 30335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:09.288424 30335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.1
I20260811 22:25:09.303669 30335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:44007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab
--principal=kudu/127.29.106.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.106.1
--webserver_port=0
--tserver_master_addrs=127.29.106.62:38933
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.1
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:25:09.305234 30335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:09.306928 30335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:09.326665 30344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:09.327299 30346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:09.329094 30341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:09.332072 30335 server_base.cc:1034] running on GCE node
I20260811 22:25:09.332710 30335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:09.333750 30335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:09.335035 30335 hybrid_clock.cc:648] HybridClock initialized: now 1786487109334972 us; error 32 us; skew 500 ppm
I20260811 22:25:09.335537 30335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:25:09 dist-test-slave-gjrz krb5kdc[30134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487109, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:09.340328 30335 init.cc:377] Logged in from keytab as kudu/127.29.106.1@KRBTEST.COM (short username kudu)
I20260811 22:25:09.342442 30335 webserver.cc:460] Webserver started at http://127.29.106.1:45869/ using document root <none> and password file <none>
I20260811 22:25:09.343411 30335 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:09.343603 30335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:09.343936 30335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:09.347695 30335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/ts-0/data/instance:
uuid: "75f6db596feb49d78bf405ff00d1a5a9"
format_stamp: "Formatted at 2026-08-11 22:25:09 on dist-test-slave-gjrz"
server_key: "276adc35e14ea0eb804cf511acfe8585"
server_key_iv: "d3418b2a71bf1301b483ecbe13b78da6"
server_key_version: "encryptionkey@0"
I20260811 22:25:09.348681 30335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance:
uuid: "75f6db596feb49d78bf405ff00d1a5a9"
format_stamp: "Formatted at 2026-08-11 22:25:09 on dist-test-slave-gjrz"
server_key: "276adc35e14ea0eb804cf511acfe8585"
server_key_iv: "d3418b2a71bf1301b483ecbe13b78da6"
server_key_version: "encryptionkey@0"
I20260811 22:25:09.355798 30335 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260811 22:25:09.361174 30358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:09.362677 30335 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260811 22:25:09.362927 30335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/ts-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/ts-0/wal
uuid: "75f6db596feb49d78bf405ff00d1a5a9"
format_stamp: "Formatted at 2026-08-11 22:25:09 on dist-test-slave-gjrz"
server_key: "276adc35e14ea0eb804cf511acfe8585"
server_key_iv: "d3418b2a71bf1301b483ecbe13b78da6"
server_key_version: "encryptionkey@0"
I20260811 22:25:09.363209 30335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:09.379492 30335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:09.385358 30335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:09.385882 30335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:09.387773 30335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:25:09.390226 30335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:25:09.390393 30335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:09.390668 30335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:25:09.390808 30335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:09.486423 30335 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.1:34229
I20260811 22:25:09.486639 30579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.1:34229 every 8 connection(s)
I20260811 22:25:09.488013 30335 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
I20260811 22:25:09.489152 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 30335
I20260811 22:25:09.489494 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance
I20260811 22:25:09.495366 30120 external_mini_cluster.cc:1428] Setting key 0d40f61fcb648ac1aa66df3b86d4afaf
Aug 11 22:25:09 dist-test-slave-gjrz krb5kdc[30134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487109, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.1@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.106.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.2@KRBTEST.COM" created.
I20260811 22:25:09.523192 30584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:09.492608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:37443 (local address 127.29.106.62:38933)
0811 22:25:09.493718 (+ 1110us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:09.493726 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:09.495816 (+ 2090us) server_negotiation.cc:407] Connection header received
0811 22:25:09.499337 (+ 3521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:09.499343 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:09.499495 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:09.499608 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:09.502610 (+ 3002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:09.504328 (+ 1718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:09.507143 (+ 2815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:09.507815 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:09.507919 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:09.513643 (+ 5724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:09.513652 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:09.513658 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:09.513706 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:09.518917 (+ 5211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:09.519845 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:09.519849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:09.519851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:09.520082 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:09.520751 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:09.520755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:09.520757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:09.521192 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:09.521348 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:09.522450 (+ 1102us) server_negotiation.cc:299] Negotiation successful
0811 22:25:09.522687 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":922,"thread_start_us":747,"threads_started":1}
I20260811 22:25:09.524835 30585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:09.493140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:38933 (local address 127.29.106.1:37443)
0811 22:25:09.495477 (+ 2337us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:09.495528 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:09.498996 (+ 3468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:09.500727 (+ 1731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:09.500741 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:09.501525 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:09.502371 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:09.502393 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:09.504544 (+ 2151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:09.504550 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:09.506988 (+ 2438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:09.507006 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:09.508196 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:09.508207 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:09.508648 (+ 441us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:09.509539 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:09.509605 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:09.513308 (+ 3703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:09.519070 (+ 5762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:09.519078 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:09.519092 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:09.519725 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:09.520195 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:09.520198 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:09.520201 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:09.520640 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:09.521368 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:09.521374 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:09.521856 (+ 482us) client_negotiation.cc:769] Sending connection context
0811 22:25:09.522360 (+ 504us) client_negotiation.cc:240] Negotiation successful
0811 22:25:09.522995 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1758,"thread_start_us":1196,"threads_started":1}
I20260811 22:25:09.526782 30582 heartbeater.cc:347] Connected to a master server at 127.29.106.62:38933
I20260811 22:25:09.527247 30582 heartbeater.cc:464] Registering TS with master...
I20260811 22:25:09.528481 30582 heartbeater.cc:511] Master 127.29.106.62:38933 requested a full tablet report, sending...
I20260811 22:25:09.532872 30227 ts_manager.cc:194] Registered new tserver with Master: 75f6db596feb49d78bf405ff00d1a5a9 (127.29.106.1:34229)
I20260811 22:25:09.535586 30227 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.1@KRBTEST.COM'} at 127.29.106.1:37443
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab.
Entry for principal kudu/127.29.106.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab.
I20260811 22:25:09.548386 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.2:0
--local_ip_for_outbound_sockets=127.29.106.2
--webserver_interface=127.29.106.2
--webserver_port=0
--tserver_master_addrs=127.29.106.62:38933
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab
--principal=kudu/127.29.106.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:44007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:25:10.242607 30591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:10.243032 30591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:10.243278 30591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:10.254511 30591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:10.254745 30591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:10.255007 30591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.2
I20260811 22:25:10.269354 30591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:44007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab
--principal=kudu/127.29.106.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.106.2
--webserver_port=0
--tserver_master_addrs=127.29.106.62:38933
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.2
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:25:10.270854 30591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:10.272625 30591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:10.289197 30600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:10.290351 30601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:10.290445 30604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:10.297257 30591 server_base.cc:1034] running on GCE node
I20260811 22:25:10.298007 30591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:10.299192 30591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:10.300505 30591 hybrid_clock.cc:648] HybridClock initialized: now 1786487110300371 us; error 54 us; skew 500 ppm
I20260811 22:25:10.300962 30591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:25:10 dist-test-slave-gjrz krb5kdc[30134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487110, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:10.310940 30591 init.cc:377] Logged in from keytab as kudu/127.29.106.2@KRBTEST.COM (short username kudu)
I20260811 22:25:10.313223 30591 webserver.cc:460] Webserver started at http://127.29.106.2:36013/ using document root <none> and password file <none>
I20260811 22:25:10.314157 30591 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:10.314334 30591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:10.314702 30591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:10.318490 30591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/ts-1/data/instance:
uuid: "5c61cf9f86b94f30ae6e859ff8e2bd21"
format_stamp: "Formatted at 2026-08-11 22:25:10 on dist-test-slave-gjrz"
server_key: "b546c2215a9a792de02b51691b74b725"
server_key_iv: "087a4f8437222bbedf6e3920108d9005"
server_key_version: "encryptionkey@0"
I20260811 22:25:10.319542 30591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance:
uuid: "5c61cf9f86b94f30ae6e859ff8e2bd21"
format_stamp: "Formatted at 2026-08-11 22:25:10 on dist-test-slave-gjrz"
server_key: "b546c2215a9a792de02b51691b74b725"
server_key_iv: "087a4f8437222bbedf6e3920108d9005"
server_key_version: "encryptionkey@0"
I20260811 22:25:10.326894 30591 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260811 22:25:10.332249 30620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:10.333725 30591 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260811 22:25:10.333943 30591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/ts-1/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/ts-1/wal
uuid: "5c61cf9f86b94f30ae6e859ff8e2bd21"
format_stamp: "Formatted at 2026-08-11 22:25:10 on dist-test-slave-gjrz"
server_key: "b546c2215a9a792de02b51691b74b725"
server_key_iv: "087a4f8437222bbedf6e3920108d9005"
server_key_version: "encryptionkey@0"
I20260811 22:25:10.334229 30591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:10.354609 30591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:10.360785 30591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:10.361189 30591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:10.362952 30591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:25:10.365444 30591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:25:10.365593 30591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:10.365828 30591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:25:10.365959 30591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:10.448800 30591 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.2:46333
I20260811 22:25:10.448952 30835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.2:46333 every 8 connection(s)
I20260811 22:25:10.450817 30591 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
I20260811 22:25:10.452247 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 30591
I20260811 22:25:10.452572 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance
I20260811 22:25:10.453302 30120 external_mini_cluster.cc:1428] Setting key 9f6ce80b70b05307ca017b43315e9d0f
Aug 11 22:25:10 dist-test-slave-gjrz krb5kdc[30134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487110, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.2@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.106.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.3@KRBTEST.COM" created.
I20260811 22:25:10.482618 30840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:10.455868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:46397 (local address 127.29.106.62:38933)
0811 22:25:10.456841 (+ 973us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:10.456845 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:10.458591 (+ 1746us) server_negotiation.cc:407] Connection header received
0811 22:25:10.461930 (+ 3339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:10.461934 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:10.462004 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:10.462103 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:10.464746 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:10.465839 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:10.468190 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:10.469028 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:10.469152 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:10.474065 (+ 4913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:10.474070 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:10.474074 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:10.474115 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:10.478694 (+ 4579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:10.479633 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:10.479638 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:10.479642 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:10.479868 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:10.480415 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:10.480420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:10.480423 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:10.480945 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:10.481113 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:10.481995 (+ 882us) server_negotiation.cc:299] Negotiation successful
0811 22:25:10.482253 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":873,"thread_start_us":641,"threads_started":1}
I20260811 22:25:10.484072 30841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:10.456624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:38933 (local address 127.29.106.2:46397)
0811 22:25:10.458401 (+ 1777us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:10.458464 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:10.461602 (+ 3138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:10.462421 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:10.462434 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:10.463497 (+ 1063us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:10.464554 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:10.464577 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:10.466007 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:10.466011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:10.468038 (+ 2027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:10.468050 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:10.469321 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:10.469330 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:10.469673 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:10.470510 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:10.470591 (+ 81us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:10.473764 (+ 3173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:10.478838 (+ 5074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:10.478846 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:10.478858 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:10.479486 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:10.479970 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:10.479974 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:10.479976 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:10.480286 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:10.481064 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:10.481070 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:10.481545 (+ 475us) client_negotiation.cc:769] Sending connection context
0811 22:25:10.481834 (+ 289us) client_negotiation.cc:240] Negotiation successful
0811 22:25:10.482559 (+ 725us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1408,"thread_start_us":991,"threads_started":1}
I20260811 22:25:10.486050 30838 heartbeater.cc:347] Connected to a master server at 127.29.106.62:38933
I20260811 22:25:10.486347 30838 heartbeater.cc:464] Registering TS with master...
I20260811 22:25:10.487044 30838 heartbeater.cc:511] Master 127.29.106.62:38933 requested a full tablet report, sending...
I20260811 22:25:10.488662 30227 ts_manager.cc:194] Registered new tserver with Master: 5c61cf9f86b94f30ae6e859ff8e2bd21 (127.29.106.2:46333)
I20260811 22:25:10.489629 30227 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.2@KRBTEST.COM'} at 127.29.106.2:46397
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab.
Entry for principal kudu/127.29.106.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab.
I20260811 22:25:10.503501 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.3:0
--local_ip_for_outbound_sockets=127.29.106.3
--webserver_interface=127.29.106.3
--webserver_port=0
--tserver_master_addrs=127.29.106.62:38933
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab
--principal=kudu/127.29.106.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:44007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 22:25:10.539479 30582 heartbeater.cc:503] Master 127.29.106.62:38933 was elected leader, sending a full tablet report...
W20260811 22:25:11.148955 30846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:11.149303 30846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:11.149515 30846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:11.160198 30846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:11.160424 30846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:11.160692 30846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.3
I20260811 22:25:11.174546 30846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:44007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab
--principal=kudu/127.29.106.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.106.3
--webserver_port=0
--tserver_master_addrs=127.29.106.62:38933
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.3
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:25:11.176045 30846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:11.177592 30846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:11.193461 30852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:11.194839 30857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:11.194957 30853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:11.196331 30846 server_base.cc:1034] running on GCE node
I20260811 22:25:11.197211 30846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:11.198514 30846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:11.199846 30846 hybrid_clock.cc:648] HybridClock initialized: now 1786487111199767 us; error 24 us; skew 500 ppm
I20260811 22:25:11.200367 30846 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:25:11 dist-test-slave-gjrz krb5kdc[30134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487111, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:11.206151 30846 init.cc:377] Logged in from keytab as kudu/127.29.106.3@KRBTEST.COM (short username kudu)
I20260811 22:25:11.208424 30846 webserver.cc:460] Webserver started at http://127.29.106.3:36063/ using document root <none> and password file <none>
I20260811 22:25:11.209389 30846 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:11.209633 30846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:11.210230 30846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:11.213967 30846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/ts-2/data/instance:
uuid: "4c5c60955b1247829bd77237342d3473"
format_stamp: "Formatted at 2026-08-11 22:25:11 on dist-test-slave-gjrz"
server_key: "03d6339d683f5957af6d0fcff741f610"
server_key_iv: "277e3404d3ba353f350d199228aadb7d"
server_key_version: "encryptionkey@0"
I20260811 22:25:11.214922 30846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance:
uuid: "4c5c60955b1247829bd77237342d3473"
format_stamp: "Formatted at 2026-08-11 22:25:11 on dist-test-slave-gjrz"
server_key: "03d6339d683f5957af6d0fcff741f610"
server_key_iv: "277e3404d3ba353f350d199228aadb7d"
server_key_version: "encryptionkey@0"
I20260811 22:25:11.222401 30846 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260811 22:25:11.227391 30866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:11.228817 30846 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260811 22:25:11.229050 30846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/ts-2/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/ts-2/wal
uuid: "4c5c60955b1247829bd77237342d3473"
format_stamp: "Formatted at 2026-08-11 22:25:11 on dist-test-slave-gjrz"
server_key: "03d6339d683f5957af6d0fcff741f610"
server_key_iv: "277e3404d3ba353f350d199228aadb7d"
server_key_version: "encryptionkey@0"
I20260811 22:25:11.229321 30846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:11.254318 30846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:11.260699 30846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:11.261198 30846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:11.263252 30846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:25:11.265672 30846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:25:11.265861 30846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:11.266111 30846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:25:11.266258 30846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:11.363101 30846 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.3:43501
I20260811 22:25:11.363425 31077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.3:43501 every 8 connection(s)
I20260811 22:25:11.364799 30846 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
I20260811 22:25:11.366302 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 30846
I20260811 22:25:11.366628 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance
I20260811 22:25:11.367377 30120 external_mini_cluster.cc:1428] Setting key 29fc19b74215737d854725e5dd6bdc3a
Aug 11 22:25:11 dist-test-slave-gjrz krb5kdc[30134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487111, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.3@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:25:11.395337 31083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:11.369087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:40673 (local address 127.29.106.62:38933)
0811 22:25:11.369774 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:11.369779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:11.372007 (+ 2228us) server_negotiation.cc:407] Connection header received
0811 22:25:11.375008 (+ 3001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:11.375012 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:11.375123 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:11.375407 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:11.377656 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:11.378689 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:11.380978 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:11.382034 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:11.382161 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:11.387798 (+ 5637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:11.387803 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:11.387807 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:11.387838 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:11.392033 (+ 4195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:11.392838 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:11.392843 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:11.392845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:11.393064 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:11.393623 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:11.393627 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:11.393629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:11.394000 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:11.394173 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:11.394775 (+ 602us) server_negotiation.cc:299] Negotiation successful
0811 22:25:11.394958 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":413,"threads_started":1}
I20260811 22:25:11.396515 31084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:11.369691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:38933 (local address 127.29.106.3:40673)
0811 22:25:11.371660 (+ 1969us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:11.371720 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:11.374770 (+ 3050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:11.375683 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:11.375691 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:11.376437 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:11.377483 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:11.377506 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:11.378888 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:11.378892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:11.380832 (+ 1940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:11.380847 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:11.382361 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:11.382369 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:11.382711 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:11.383526 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:11.383561 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:11.387519 (+ 3958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:11.392175 (+ 4656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:11.392182 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:11.392192 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:11.392714 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:11.393152 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:11.393155 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:11.393158 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:11.393473 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:11.394121 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:11.394126 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:11.394438 (+ 312us) client_negotiation.cc:769] Sending connection context
0811 22:25:11.394697 (+ 259us) client_negotiation.cc:240] Negotiation successful
0811 22:25:11.395100 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1512,"thread_start_us":1109,"threads_started":1}
I20260811 22:25:11.398216 31078 heartbeater.cc:347] Connected to a master server at 127.29.106.62:38933
I20260811 22:25:11.398571 31078 heartbeater.cc:464] Registering TS with master...
I20260811 22:25:11.399654 31078 heartbeater.cc:511] Master 127.29.106.62:38933 requested a full tablet report, sending...
I20260811 22:25:11.401332 30227 ts_manager.cc:194] Registered new tserver with Master: 4c5c60955b1247829bd77237342d3473 (127.29.106.3:43501)
I20260811 22:25:11.402266 30227 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.3@KRBTEST.COM'} at 127.29.106.3:40673
I20260811 22:25:11.409879 30120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 11 22:25:11 dist-test-slave-gjrz krb5kdc[30134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487111, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 11 22:25:11 dist-test-slave-gjrz krb5kdc[30134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487111, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.106.1@KRBTEST.COM
I20260811 22:25:11.469174 31093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:11.450731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37278 (local address 127.29.106.1:34229)
0811 22:25:11.452067 (+ 1336us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:11.452075 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:11.452102 (+ 27us) server_negotiation.cc:407] Connection header received
0811 22:25:11.452161 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:11.452167 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:11.452373 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:11.452493 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:11.454053 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:11.455528 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:11.456766 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:11.457315 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:11.457411 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:11.461256 (+ 3845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:11.461268 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:11.461281 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:11.461313 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:11.465624 (+ 4311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:11.466476 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:11.466485 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:11.466489 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:11.466699 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:11.467230 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:11.467233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:11.467235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:11.467748 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:11.468174 (+ 426us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:11.468332 (+ 158us) server_negotiation.cc:299] Negotiation successful
0811 22:25:11.468561 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1115,"thread_start_us":825,"threads_started":1}
W20260811 22:25:11.470846 30413 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37278
I20260811 22:25:11.492479 30838 heartbeater.cc:503] Master 127.29.106.62:38933 was elected leader, sending a full tablet report...
Aug 11 22:25:11 dist-test-slave-gjrz krb5kdc[30134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 11 22:25:11 dist-test-slave-gjrz krb5kdc[30134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.1@KRBTEST.COM
I20260811 22:25:11.522017 31093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:11.505089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37290 (local address 127.29.106.1:34229)
0811 22:25:11.505295 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:11.505302 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:11.505832 (+ 530us) server_negotiation.cc:407] Connection header received
0811 22:25:11.506012 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:11.506017 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:11.506080 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:11.506159 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:11.508593 (+ 2434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:11.509498 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:11.511371 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:11.511796 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:11.511900 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:11.516948 (+ 5048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:11.516953 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:11.516956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:11.516990 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:11.519524 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:11.520133 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:11.520137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:11.520139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:11.520270 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:11.520617 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:11.520621 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:11.520623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:11.520907 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:11.521136 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:11.521451 (+ 315us) server_negotiation.cc:299] Negotiation successful
0811 22:25:11.521704 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260811 22:25:11.524003 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 30335
I20260811 22:25:11.541275 31093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:11.525121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37302 (local address 127.29.106.1:34229)
0811 22:25:11.525293 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:11.525298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:11.525944 (+ 646us) server_negotiation.cc:407] Connection header received
0811 22:25:11.526061 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:11.526064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:11.526114 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:11.526188 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:11.528157 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:11.528866 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:11.530395 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:11.530807 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:11.530912 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:11.534252 (+ 3340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:11.534256 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:11.534260 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:11.534288 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:11.538257 (+ 3969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:11.539042 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:11.539046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:11.539048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:11.539237 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:11.539734 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:11.539740 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:11.539742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:11.540030 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:11.540315 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:11.540544 (+ 229us) server_negotiation.cc:299] Negotiation successful
0811 22:25:11.540736 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260811 22:25:11.542474 30564 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:25:11.631335 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 30335
I20260811 22:25:11.651414 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 30591
Aug 11 22:25:11 dist-test-slave-gjrz krb5kdc[30134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.2@KRBTEST.COM
I20260811 22:25:11.669775 31104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:11.652270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41640 (local address 127.29.106.2:46333)
0811 22:25:11.652855 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:11.652861 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:11.652886 (+ 25us) server_negotiation.cc:407] Connection header received
0811 22:25:11.652947 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:11.652954 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:11.653171 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:11.653294 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:11.655212 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:11.656391 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:11.658230 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:11.658793 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:11.658883 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:11.664445 (+ 5562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:11.664456 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:11.664470 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:11.664511 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:11.667016 (+ 2505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:11.667822 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:11.667829 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:11.667831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:11.667955 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:11.668400 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:11.668403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:11.668405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:11.668723 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:11.668941 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:11.669212 (+ 271us) server_negotiation.cc:299] Negotiation successful
0811 22:25:11.669427 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":272,"threads_started":1}
I20260811 22:25:11.671211 30822 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:25:11.751940 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 30591
I20260811 22:25:11.779395 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 30846
Aug 11 22:25:11 dist-test-slave-gjrz krb5kdc[30134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.3@KRBTEST.COM
I20260811 22:25:11.795652 31107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:11.780538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60752 (local address 127.29.106.3:43501)
0811 22:25:11.781164 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:11.781172 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:11.781203 (+ 31us) server_negotiation.cc:407] Connection header received
0811 22:25:11.781267 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:11.781274 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:11.781504 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:11.781657 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:11.782996 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:11.784107 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:11.785387 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:11.785952 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:11.786068 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:11.790186 (+ 4118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:11.790198 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:11.790213 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:11.790254 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:11.792889 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:11.793513 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:11.793520 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:11.793523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:11.793662 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:11.794027 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:11.794031 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:11.794034 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:11.794389 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:11.794689 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:11.795003 (+ 314us) server_negotiation.cc:299] Negotiation successful
0811 22:25:11.795303 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":299,"threads_started":1}
I20260811 22:25:11.797175 31071 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:25:11.878855 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 30846
I20260811 22:25:11.908433 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 30162
I20260811 22:25:11.909580 30278 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:25:12.010074 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 30162
2026-08-11T22:25:12Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (6119 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:25:12 dist-test-slave-gjrz krb5kdc[31115](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:25:12 dist-test-slave-gjrz krb5kdc[31115](info): set up 2 sockets
Aug 11 22:25:12 dist-test-slave-gjrz krb5kdc[31115](info): commencing operation
krb5kdc: starting...
W20260811 22:25:13.261469 30120 mini_kdc.cc:121] Time spent starting KDC: real 1.206s user 0.001s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Aug 11 22:25:13 dist-test-slave-gjrz krb5kdc[31115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:25:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:25:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.106.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
I20260811 22:25:13.424391 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:36361
--webserver_interface=127.29.106.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:45595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:36361
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:25:13.740067 31142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:13.740337 31142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:13.740478 31142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:13.747452 31142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:25:13.747556 31142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:13.747635 31142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:13.747716 31142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:25:13.747777 31142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:25:13.757028 31142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:45595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:36361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:36361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.106.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:25:13.758210 31142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:13.759471 31142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:13.770568 31157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:13.770640 31152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:13.770612 31153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:13.771533 31142 server_base.cc:1034] running on GCE node
I20260811 22:25:13.772199 31142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:13.773545 31142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:13.774760 31142 hybrid_clock.cc:648] HybridClock initialized: now 1786487113774707 us; error 49 us; skew 500 ppm
I20260811 22:25:13.775110 31142 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:25:13 dist-test-slave-gjrz krb5kdc[31115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487113, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:13.778178 31142 init.cc:377] Logged in from keytab as kudu/127.29.106.62@KRBTEST.COM (short username kudu)
I20260811 22:25:13.779524 31142 webserver.cc:460] Webserver started at http://127.29.106.62:33277/ using document root <none> and password file <none>
I20260811 22:25:13.780115 31142 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:13.780222 31142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:13.780488 31142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:13.783200 31142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "adb8fa25fe5b4bff80fb78a9029399ec"
format_stamp: "Formatted at 2026-08-11 22:25:13 on dist-test-slave-gjrz"
server_key: "f53243185543f9fb2da78a035d221889"
server_key_iv: "accd13357eb63037fba0a03f412f37bd"
server_key_version: "encryptionkey@0"
I20260811 22:25:13.783927 31142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "adb8fa25fe5b4bff80fb78a9029399ec"
format_stamp: "Formatted at 2026-08-11 22:25:13 on dist-test-slave-gjrz"
server_key: "f53243185543f9fb2da78a035d221889"
server_key_iv: "accd13357eb63037fba0a03f412f37bd"
server_key_version: "encryptionkey@0"
I20260811 22:25:13.788822 31142 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260811 22:25:13.792057 31166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:13.793224 31142 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260811 22:25:13.793355 31142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "adb8fa25fe5b4bff80fb78a9029399ec"
format_stamp: "Formatted at 2026-08-11 22:25:13 on dist-test-slave-gjrz"
server_key: "f53243185543f9fb2da78a035d221889"
server_key_iv: "accd13357eb63037fba0a03f412f37bd"
server_key_version: "encryptionkey@0"
I20260811 22:25:13.793504 31142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:13.820104 31142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:13.824347 31142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:13.824610 31142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:13.841733 31142 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.62:36361
I20260811 22:25:13.841756 31257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.62:36361 every 8 connection(s)
I20260811 22:25:13.843021 31142 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:25:13.846674 31259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:13.848758 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 31142
I20260811 22:25:13.848963 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:25:13.849541 30120 external_mini_cluster.cc:1428] Setting key df1869327f69d3d1078da029770832a3
I20260811 22:25:13.859187 31259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adb8fa25fe5b4bff80fb78a9029399ec: Bootstrap starting.
I20260811 22:25:13.862607 31259 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adb8fa25fe5b4bff80fb78a9029399ec: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:13.863631 31259 log.cc:824] T 00000000000000000000000000000000 P adb8fa25fe5b4bff80fb78a9029399ec: Log is configured to *not* fsync() on all Append() calls
Aug 11 22:25:13 dist-test-slave-gjrz krb5kdc[31115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:25:13.866281 31259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adb8fa25fe5b4bff80fb78a9029399ec: No bootstrap required, opened a new log
I20260811 22:25:13.871745 31266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:13.851417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35026 (local address 127.29.106.62:36361)
0811 22:25:13.852242 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:13.852257 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:13.852313 (+ 56us) server_negotiation.cc:407] Connection header received
0811 22:25:13.854613 (+ 2300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:13.854641 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:13.855094 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:13.855549 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:13.856800 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:13.858707 (+ 1907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:13.859957 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:13.860997 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:13.861182 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:13.864915 (+ 3733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:13.864923 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:13.864937 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:13.864974 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:13.867801 (+ 2827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:13.868555 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:13.868562 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:13.868567 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:13.868732 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:13.869090 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:13.869093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:13.869095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:13.869593 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:13.869810 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:13.870325 (+ 515us) server_negotiation.cc:299] Negotiation successful
0811 22:25:13.870655 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":337,"threads_started":1}
I20260811 22:25:13.875168 31259 raft_consensus.cc:348] T 00000000000000000000000000000000 P adb8fa25fe5b4bff80fb78a9029399ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb8fa25fe5b4bff80fb78a9029399ec" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 36361 } }
I20260811 22:25:13.875524 31259 raft_consensus.cc:374] T 00000000000000000000000000000000 P adb8fa25fe5b4bff80fb78a9029399ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:25:13.875617 31259 raft_consensus.cc:729] T 00000000000000000000000000000000 P adb8fa25fe5b4bff80fb78a9029399ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adb8fa25fe5b4bff80fb78a9029399ec, State: Initialized, Role: FOLLOWER
I20260811 22:25:13.876113 31259 consensus_queue.cc:260] T 00000000000000000000000000000000 P adb8fa25fe5b4bff80fb78a9029399ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb8fa25fe5b4bff80fb78a9029399ec" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 36361 } }
I20260811 22:25:13.876338 31259 raft_consensus.cc:388] T 00000000000000000000000000000000 P adb8fa25fe5b4bff80fb78a9029399ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:25:13.876457 31259 raft_consensus.cc:482] T 00000000000000000000000000000000 P adb8fa25fe5b4bff80fb78a9029399ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:25:13.876605 31259 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adb8fa25fe5b4bff80fb78a9029399ec [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:13.878818 31259 raft_consensus.cc:504] T 00000000000000000000000000000000 P adb8fa25fe5b4bff80fb78a9029399ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb8fa25fe5b4bff80fb78a9029399ec" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 36361 } }
I20260811 22:25:13.879330 31259 leader_election.cc:302] T 00000000000000000000000000000000 P adb8fa25fe5b4bff80fb78a9029399ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: adb8fa25fe5b4bff80fb78a9029399ec; no voters:
I20260811 22:25:13.879842 31259 leader_election.cc:288] T 00000000000000000000000000000000 P adb8fa25fe5b4bff80fb78a9029399ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:25:13.879948 31268 raft_consensus.cc:2781] T 00000000000000000000000000000000 P adb8fa25fe5b4bff80fb78a9029399ec [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:25:13.881153 31268 raft_consensus.cc:686] T 00000000000000000000000000000000 P adb8fa25fe5b4bff80fb78a9029399ec [term 1 LEADER]: Becoming Leader. State: Replica: adb8fa25fe5b4bff80fb78a9029399ec, State: Running, Role: LEADER
I20260811 22:25:13.881709 31268 consensus_queue.cc:237] T 00000000000000000000000000000000 P adb8fa25fe5b4bff80fb78a9029399ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb8fa25fe5b4bff80fb78a9029399ec" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 36361 } }
I20260811 22:25:13.881820 31259 sys_catalog.cc:564] T 00000000000000000000000000000000 P adb8fa25fe5b4bff80fb78a9029399ec [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:25:13.884872 31272 sys_catalog.cc:455] T 00000000000000000000000000000000 P adb8fa25fe5b4bff80fb78a9029399ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader adb8fa25fe5b4bff80fb78a9029399ec. Latest consensus state: current_term: 1 leader_uuid: "adb8fa25fe5b4bff80fb78a9029399ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb8fa25fe5b4bff80fb78a9029399ec" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 36361 } } }
I20260811 22:25:13.885432 31272 sys_catalog.cc:458] T 00000000000000000000000000000000 P adb8fa25fe5b4bff80fb78a9029399ec [sys.catalog]: This master's current role is: LEADER
I20260811 22:25:13.885010 31271 sys_catalog.cc:455] T 00000000000000000000000000000000 P adb8fa25fe5b4bff80fb78a9029399ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "adb8fa25fe5b4bff80fb78a9029399ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb8fa25fe5b4bff80fb78a9029399ec" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 36361 } } }
I20260811 22:25:13.886117 31271 sys_catalog.cc:458] T 00000000000000000000000000000000 P adb8fa25fe5b4bff80fb78a9029399ec [sys.catalog]: This master's current role is: LEADER
I20260811 22:25:13.886951 31284 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:25:13.891728 31284 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:25:13.900319 31284 catalog_manager.cc:1347] Generated new cluster ID: 40dc4343c4174186898b0d7450c50ae6
I20260811 22:25:13.900450 31284 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:25:13.913554 31284 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:25:13.914880 31284 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:25:13.923988 31284 catalog_manager.cc:5868] T 00000000000000000000000000000000 P adb8fa25fe5b4bff80fb78a9029399ec: Generated new TSK 0
I20260811 22:25:13.924832 31284 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.106.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab.
Entry for principal kudu/127.29.106.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab.
I20260811 22:25:13.976963 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.1:0
--local_ip_for_outbound_sockets=127.29.106.1
--webserver_interface=127.29.106.1
--webserver_port=0
--tserver_master_addrs=127.29.106.62:36361
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab
--principal=kudu/127.29.106.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:45595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:25:14.318128 31300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:14.318375 31300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:14.318499 31300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:14.325660 31300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:14.325762 31300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:14.325922 31300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.1
I20260811 22:25:14.335364 31300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:45595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab
--principal=kudu/127.29.106.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.106.1
--webserver_port=0
--tserver_master_addrs=127.29.106.62:36361
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.1
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:25:14.336555 31300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:14.337821 31300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:14.350059 31310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:14.350118 31314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:14.350080 31311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:14.350919 31300 server_base.cc:1034] running on GCE node
I20260811 22:25:14.351543 31300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:14.352443 31300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:14.353652 31300 hybrid_clock.cc:648] HybridClock initialized: now 1786487114353599 us; error 44 us; skew 500 ppm
I20260811 22:25:14.353974 31300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:25:14 dist-test-slave-gjrz krb5kdc[31115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487114, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:14.357088 31300 init.cc:377] Logged in from keytab as kudu/127.29.106.1@KRBTEST.COM (short username kudu)
I20260811 22:25:14.358508 31300 webserver.cc:460] Webserver started at http://127.29.106.1:41493/ using document root <none> and password file <none>
I20260811 22:25:14.359169 31300 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:14.359314 31300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:14.359616 31300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:14.362444 31300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/ts-0/data/instance:
uuid: "a40ed35af46f4b048d1ecfe15f291176"
format_stamp: "Formatted at 2026-08-11 22:25:14 on dist-test-slave-gjrz"
server_key: "56b44b88f57954a779b1d3a9d706ea3e"
server_key_iv: "0951dfab721d0808f80b750b172b84cf"
server_key_version: "encryptionkey@0"
I20260811 22:25:14.363189 31300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance:
uuid: "a40ed35af46f4b048d1ecfe15f291176"
format_stamp: "Formatted at 2026-08-11 22:25:14 on dist-test-slave-gjrz"
server_key: "56b44b88f57954a779b1d3a9d706ea3e"
server_key_iv: "0951dfab721d0808f80b750b172b84cf"
server_key_version: "encryptionkey@0"
I20260811 22:25:14.368288 31300 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260811 22:25:14.371721 31323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:14.372989 31300 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260811 22:25:14.373152 31300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/ts-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/ts-0/wal
uuid: "a40ed35af46f4b048d1ecfe15f291176"
format_stamp: "Formatted at 2026-08-11 22:25:14 on dist-test-slave-gjrz"
server_key: "56b44b88f57954a779b1d3a9d706ea3e"
server_key_iv: "0951dfab721d0808f80b750b172b84cf"
server_key_version: "encryptionkey@0"
I20260811 22:25:14.373327 31300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:14.384826 31300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:14.388911 31300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:14.389178 31300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:14.390033 31300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:25:14.391245 31300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:25:14.391371 31300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:14.391486 31300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:25:14.391539 31300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:14.421559 31300 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.1:38321
I20260811 22:25:14.421604 31528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.1:38321 every 8 connection(s)
I20260811 22:25:14.422766 31300 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
I20260811 22:25:14.425055 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 31300
I20260811 22:25:14.425263 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance
I20260811 22:25:14.425972 30120 external_mini_cluster.cc:1428] Setting key 7c9e61a2df537e8d539bf983fd2cc014
Aug 11 22:25:14 dist-test-slave-gjrz krb5kdc[31115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487114, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.1@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:25:14.444185 31533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:14.425342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:53289 (local address 127.29.106.62:36361)
0811 22:25:14.425812 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:14.425819 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:14.427124 (+ 1305us) server_negotiation.cc:407] Connection header received
0811 22:25:14.429129 (+ 2005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:14.429133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:14.429203 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:14.429289 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:14.431225 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:14.431986 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:14.433982 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:14.434559 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:14.434674 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:14.437834 (+ 3160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:14.437840 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:14.437844 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:14.437881 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:14.441082 (+ 3201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:14.441971 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:14.441974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:14.441976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:14.442089 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:14.442532 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:14.442535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:14.442537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:14.442782 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:14.442888 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:14.443744 (+ 856us) server_negotiation.cc:299] Negotiation successful
0811 22:25:14.443907 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":222,"threads_started":1}
WARNING: no policy specified for kudu/127.29.106.2@KRBTEST.COM; defaulting to no policy
I20260811 22:25:14.445462 31535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:14.425862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:36361 (local address 127.29.106.1:53289)
0811 22:25:14.426697 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:14.426739 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:14.428849 (+ 2110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:14.429570 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:14.429582 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:14.430281 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:14.431047 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:14.431074 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:14.432137 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:14.432142 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:14.433829 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:14.433850 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:14.434760 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:14.434764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:14.434956 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:14.435598 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:14.435619 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:14.437626 (+ 2007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:14.441214 (+ 3588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:14.441223 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:14.441241 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:14.441839 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:14.442187 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:14.442192 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:14.442196 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:14.442440 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:14.442894 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:14.442902 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:14.443367 (+ 465us) client_negotiation.cc:769] Sending connection context
0811 22:25:14.443681 (+ 314us) client_negotiation.cc:240] Negotiation successful
0811 22:25:14.444200 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":309,"threads_started":1}
I20260811 22:25:14.446900 31529 heartbeater.cc:347] Connected to a master server at 127.29.106.62:36361
I20260811 22:25:14.447175 31529 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.2@KRBTEST.COM" created.
I20260811 22:25:14.447912 31529 heartbeater.cc:511] Master 127.29.106.62:36361 requested a full tablet report, sending...
I20260811 22:25:14.450172 31190 ts_manager.cc:194] Registered new tserver with Master: a40ed35af46f4b048d1ecfe15f291176 (127.29.106.1:38321)
I20260811 22:25:14.451683 31190 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.1@KRBTEST.COM'} at 127.29.106.1:53289
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab.
Entry for principal kudu/127.29.106.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab.
I20260811 22:25:14.468561 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.2:0
--local_ip_for_outbound_sockets=127.29.106.2
--webserver_interface=127.29.106.2
--webserver_port=0
--tserver_master_addrs=127.29.106.62:36361
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab
--principal=kudu/127.29.106.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:45595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:25:14.790630 31538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:14.790913 31538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:14.791061 31538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:14.800905 31538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:14.801041 31538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:14.801234 31538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.2
I20260811 22:25:14.815222 31538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:45595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab
--principal=kudu/127.29.106.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.106.2
--webserver_port=0
--tserver_master_addrs=127.29.106.62:36361
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.2
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:25:14.816816 31538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:14.818516 31538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:14.831800 31547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:14.831820 31549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:14.833092 31538 server_base.cc:1034] running on GCE node
W20260811 22:25:14.833133 31553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:14.833933 31538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:14.835018 31538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:14.836259 31538 hybrid_clock.cc:648] HybridClock initialized: now 1786487114836177 us; error 73 us; skew 500 ppm
I20260811 22:25:14.836566 31538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:25:14 dist-test-slave-gjrz krb5kdc[31115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487114, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:14.839596 31538 init.cc:377] Logged in from keytab as kudu/127.29.106.2@KRBTEST.COM (short username kudu)
I20260811 22:25:14.840947 31538 webserver.cc:460] Webserver started at http://127.29.106.2:42943/ using document root <none> and password file <none>
I20260811 22:25:14.841655 31538 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:14.841768 31538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:14.842079 31538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:14.844874 31538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/ts-1/data/instance:
uuid: "636c2e1112304325a775747db1320a92"
format_stamp: "Formatted at 2026-08-11 22:25:14 on dist-test-slave-gjrz"
server_key: "de4673c2adf650ad908fe116a033480d"
server_key_iv: "afd0eec7bdeddbce968c9a5b51e18e77"
server_key_version: "encryptionkey@0"
I20260811 22:25:14.845623 31538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance:
uuid: "636c2e1112304325a775747db1320a92"
format_stamp: "Formatted at 2026-08-11 22:25:14 on dist-test-slave-gjrz"
server_key: "de4673c2adf650ad908fe116a033480d"
server_key_iv: "afd0eec7bdeddbce968c9a5b51e18e77"
server_key_version: "encryptionkey@0"
I20260811 22:25:14.850646 31538 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260811 22:25:14.854067 31562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:14.855463 31538 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260811 22:25:14.855608 31538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/ts-1/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/ts-1/wal
uuid: "636c2e1112304325a775747db1320a92"
format_stamp: "Formatted at 2026-08-11 22:25:14 on dist-test-slave-gjrz"
server_key: "de4673c2adf650ad908fe116a033480d"
server_key_iv: "afd0eec7bdeddbce968c9a5b51e18e77"
server_key_version: "encryptionkey@0"
I20260811 22:25:14.855779 31538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:14.878612 31538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:14.882328 31538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:14.882576 31538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:14.883524 31538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:25:14.884739 31538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:25:14.884811 31538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:14.884934 31538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:25:14.884974 31538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:14.912770 31538 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.2:46477
I20260811 22:25:14.912835 31774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.2:46477 every 8 connection(s)
I20260811 22:25:14.913928 31538 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
I20260811 22:25:14.923612 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 31538
I20260811 22:25:14.923815 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance
I20260811 22:25:14.924372 30120 external_mini_cluster.cc:1428] Setting key f46c59e887dc7a87baa5cb3c8a196227
Aug 11 22:25:14 dist-test-slave-gjrz krb5kdc[31115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487114, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.2@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:25:14.934176 31533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:14.916284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:39291 (local address 127.29.106.62:36361)
0811 22:25:14.916561 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:14.916566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:14.917754 (+ 1188us) server_negotiation.cc:407] Connection header received
0811 22:25:14.919612 (+ 1858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:14.919617 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:14.919872 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:14.920001 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:14.921473 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:14.922219 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:14.924042 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:14.924597 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:14.924702 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:14.928349 (+ 3647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:14.928354 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:14.928358 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:14.928389 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:14.930859 (+ 2470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:14.931732 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:14.931738 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:14.931741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:14.931946 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:14.932361 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:14.932366 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:14.932369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:14.932703 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:14.932844 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:14.933449 (+ 605us) server_negotiation.cc:299] Negotiation successful
0811 22:25:14.933652 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260811 22:25:14.934948 31779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:14.916853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:36361 (local address 127.29.106.2:39291)
0811 22:25:14.917564 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:14.917630 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:14.919358 (+ 1728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:14.920249 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:14.920258 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:14.920765 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:14.921297 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:14.921314 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:14.922367 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:14.922370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:14.923896 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:14.923911 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:14.924824 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:14.924832 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:14.925080 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:14.925680 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:14.925701 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:14.928135 (+ 2434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:14.930994 (+ 2859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:14.931002 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:14.931015 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:14.931570 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:14.932073 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:14.932076 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:14.932078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:14.932249 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:14.932823 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:14.932828 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:14.933100 (+ 272us) client_negotiation.cc:769] Sending connection context
0811 22:25:14.933304 (+ 204us) client_negotiation.cc:240] Negotiation successful
0811 22:25:14.933658 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":283,"threads_started":1}
I20260811 22:25:14.936695 31776 heartbeater.cc:347] Connected to a master server at 127.29.106.62:36361
I20260811 22:25:14.937057 31776 heartbeater.cc:464] Registering TS with master...
I20260811 22:25:14.938054 31776 heartbeater.cc:511] Master 127.29.106.62:36361 requested a full tablet report, sending...
I20260811 22:25:14.939852 31190 ts_manager.cc:194] Registered new tserver with Master: 636c2e1112304325a775747db1320a92 (127.29.106.2:46477)
I20260811 22:25:14.940778 31190 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.2@KRBTEST.COM'} at 127.29.106.2:39291
WARNING: no policy specified for kudu/127.29.106.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab.
Entry for principal kudu/127.29.106.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab.
I20260811 22:25:14.965837 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.3:0
--local_ip_for_outbound_sockets=127.29.106.3
--webserver_interface=127.29.106.3
--webserver_port=0
--tserver_master_addrs=127.29.106.62:36361
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab
--principal=kudu/127.29.106.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:45595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:25:15.393236 31788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:15.393486 31788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:15.393615 31788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:15.400607 31788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:15.400705 31788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:15.400852 31788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.3
I20260811 22:25:15.410036 31788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:45595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab
--principal=kudu/127.29.106.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.106.3
--webserver_port=0
--tserver_master_addrs=127.29.106.62:36361
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.3
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:25:15.411142 31788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:15.412396 31788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:15.427088 31795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:15.427109 31796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:15.428079 31800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:15.432595 31788 server_base.cc:1034] running on GCE node
I20260811 22:25:15.433159 31788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:15.434010 31788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:15.435210 31788 hybrid_clock.cc:648] HybridClock initialized: now 1786487115435169 us; error 33 us; skew 500 ppm
I20260811 22:25:15.435531 31788 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:25:15 dist-test-slave-gjrz krb5kdc[31115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487115, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:15.441738 31788 init.cc:377] Logged in from keytab as kudu/127.29.106.3@KRBTEST.COM (short username kudu)
I20260811 22:25:15.443177 31788 webserver.cc:460] Webserver started at http://127.29.106.3:34997/ using document root <none> and password file <none>
I20260811 22:25:15.443862 31788 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:15.443981 31788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:15.444274 31788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:15.447009 31788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/ts-2/data/instance:
uuid: "f36008e0e45b432092f555680bfa7104"
format_stamp: "Formatted at 2026-08-11 22:25:15 on dist-test-slave-gjrz"
server_key: "30c9caf0cffc0160ee7a79b4b5353422"
server_key_iv: "aa6048de6e0a5664c87cb88502a85cc8"
server_key_version: "encryptionkey@0"
I20260811 22:25:15.447798 31788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance:
uuid: "f36008e0e45b432092f555680bfa7104"
format_stamp: "Formatted at 2026-08-11 22:25:15 on dist-test-slave-gjrz"
server_key: "30c9caf0cffc0160ee7a79b4b5353422"
server_key_iv: "aa6048de6e0a5664c87cb88502a85cc8"
server_key_version: "encryptionkey@0"
I20260811 22:25:15.452521 31788 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 22:25:15.454504 31529 heartbeater.cc:503] Master 127.29.106.62:36361 was elected leader, sending a full tablet report...
I20260811 22:25:15.456668 31814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:15.457937 31788 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 22:25:15.458083 31788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/ts-2/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/ts-2/wal
uuid: "f36008e0e45b432092f555680bfa7104"
format_stamp: "Formatted at 2026-08-11 22:25:15 on dist-test-slave-gjrz"
server_key: "30c9caf0cffc0160ee7a79b4b5353422"
server_key_iv: "aa6048de6e0a5664c87cb88502a85cc8"
server_key_version: "encryptionkey@0"
I20260811 22:25:15.458248 31788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:15.471444 31788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:15.474934 31788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:15.475178 31788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:15.476044 31788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:25:15.477388 31788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:25:15.477464 31788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:15.477596 31788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:25:15.477653 31788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:15.533677 31788 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.3:35867
I20260811 22:25:15.533715 32014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.3:35867 every 8 connection(s)
I20260811 22:25:15.535673 31788 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
I20260811 22:25:15.541894 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 31788
I20260811 22:25:15.542105 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance
I20260811 22:25:15.542822 30120 external_mini_cluster.cc:1428] Setting key 1ae3e0dae5d62b4ac450539e9f1f1e08
Aug 11 22:25:15 dist-test-slave-gjrz krb5kdc[31115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487115, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.3@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:25:15.570667 32019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:15.544751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:36859 (local address 127.29.106.62:36361)
0811 22:25:15.545237 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:15.545243 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:15.547923 (+ 2680us) server_negotiation.cc:407] Connection header received
0811 22:25:15.550986 (+ 3063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:15.550990 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:15.551054 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:15.551140 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:15.552942 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:15.554098 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:15.556264 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:15.556913 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:15.557012 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:15.562542 (+ 5530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:15.562547 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:15.562550 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:15.562603 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:15.567067 (+ 4464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:15.567826 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:15.567830 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:15.567833 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:15.567996 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:15.568576 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:15.568579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:15.568585 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:15.568927 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:15.569077 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:15.569888 (+ 811us) server_negotiation.cc:299] Negotiation successful
0811 22:25:15.570112 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":258,"threads_started":1}
I20260811 22:25:15.571939 32021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:15.545781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:36361 (local address 127.29.106.3:36859)
0811 22:25:15.547758 (+ 1977us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:15.547802 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:15.550728 (+ 2926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:15.551399 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:15.551409 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:15.552058 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:15.552767 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:15.552789 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:15.554243 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:15.554248 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:15.556130 (+ 1882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:15.556144 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:15.557130 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:15.557136 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:15.557410 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:15.558505 (+ 1095us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:15.558546 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:15.562329 (+ 3783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:15.567195 (+ 4866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:15.567204 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:15.567218 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:15.567703 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:15.568103 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:15.568108 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:15.568112 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:15.568453 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:15.569071 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:15.569079 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:15.569514 (+ 435us) client_negotiation.cc:769] Sending connection context
0811 22:25:15.569757 (+ 243us) client_negotiation.cc:240] Negotiation successful
0811 22:25:15.570408 (+ 651us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1659,"thread_start_us":1265,"threads_started":1}
I20260811 22:25:15.574260 32017 heartbeater.cc:347] Connected to a master server at 127.29.106.62:36361
I20260811 22:25:15.574637 32017 heartbeater.cc:464] Registering TS with master...
I20260811 22:25:15.575728 32017 heartbeater.cc:511] Master 127.29.106.62:36361 requested a full tablet report, sending...
I20260811 22:25:15.577317 31190 ts_manager.cc:194] Registered new tserver with Master: f36008e0e45b432092f555680bfa7104 (127.29.106.3:35867)
I20260811 22:25:15.578383 31190 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.3@KRBTEST.COM'} at 127.29.106.3:36859
I20260811 22:25:15.583776 30120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 22:25:15.610054 32019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:15.592623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35028 (local address 127.29.106.62:36361)
0811 22:25:15.592799 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:15.592804 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:15.593833 (+ 1029us) server_negotiation.cc:407] Connection header received
0811 22:25:15.593984 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:15.593987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:15.594048 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:15.594118 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:15.597064 (+ 2946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:15.598062 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:15.599883 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:15.600576 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:15.600668 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:15.603616 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:15.603622 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:15.603625 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:15.603654 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:15.607307 (+ 3653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:15.607916 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:15.607919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:15.607922 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:15.608095 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:15.608503 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:15.608506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:15.608508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:15.608905 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:15.609161 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:15.609420 (+ 259us) server_negotiation.cc:299] Negotiation successful
0811 22:25:15.609633 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260811 22:25:15.618774 31190 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35028:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:25:15.631177 31190 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:25:15.681847 32044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:15.657970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41020 (local address 127.29.106.2:46477)
0811 22:25:15.669480 (+ 11510us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:15.669491 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:15.670222 (+ 731us) server_negotiation.cc:407] Connection header received
0811 22:25:15.670432 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:15.670442 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:15.670740 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:15.670973 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:25:15.671801 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:15.674111 (+ 2310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:15.678128 (+ 4017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:15.680250 (+ 2122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:15.680416 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:15.680667 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:15.681146 (+ 479us) server_negotiation.cc:299] Negotiation successful
0811 22:25:15.681350 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11336,"thread_start_us":11241,"threads_started":1}
I20260811 22:25:15.681962 32039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:15.657843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.2:46477 (local address 127.0.0.1:41020)
0811 22:25:15.670075 (+ 12232us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:15.670112 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:15.670257 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:15.670985 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:15.670990 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:15.671014 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:25:15.671530 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:15.671540 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:15.674239 (+ 2699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:15.674244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:15.677948 (+ 3704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:15.677961 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:15.680535 (+ 2574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:15.680542 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:15.680858 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:15.680934 (+ 76us) client_negotiation.cc:769] Sending connection context
0811 22:25:15.681063 (+ 129us) client_negotiation.cc:240] Negotiation successful
0811 22:25:15.681245 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12096,"thread_start_us":10199,"threads_started":1}
I20260811 22:25:15.682919 32037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:15.650614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:38321 (local address 127.0.0.1:51290)
0811 22:25:15.666385 (+ 15771us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:15.666461 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:15.666715 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:15.671846 (+ 5131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:15.671850 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:15.671876 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:25:15.673250 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:15.673267 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:15.675549 (+ 2282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:15.675554 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:15.678863 (+ 3309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:15.678875 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:15.681658 (+ 2783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:15.681666 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:15.681948 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:15.681982 (+ 34us) client_negotiation.cc:769] Sending connection context
0811 22:25:15.682063 (+ 81us) client_negotiation.cc:240] Negotiation successful
0811 22:25:15.682185 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15455,"mutex_wait_us":3,"thread_start_us":16275,"threads_started":1}
I20260811 22:25:15.684477 32042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:15.651088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51290 (local address 127.29.106.1:38321)
0811 22:25:15.671047 (+ 19959us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:15.671054 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:15.671084 (+ 30us) server_negotiation.cc:407] Connection header received
0811 22:25:15.671184 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:15.671191 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:15.671475 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:15.671856 (+ 381us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:25:15.673505 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:15.675430 (+ 1925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:15.679183 (+ 3753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:15.681251 (+ 2068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:15.683243 (+ 1992us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:15.683619 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:15.683699 (+ 80us) server_negotiation.cc:299] Negotiation successful
0811 22:25:15.683894 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19668,"thread_start_us":19245,"threads_started":1}
I20260811 22:25:15.684526 32048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:15.650871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.3:35867 (local address 127.0.0.1:36810)
0811 22:25:15.668530 (+ 17659us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:15.668561 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:15.668685 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:15.672461 (+ 3776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:15.672466 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:15.672493 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:25:15.673820 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:15.673833 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:15.677041 (+ 3208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:15.677046 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:15.680699 (+ 3653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:15.680713 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:15.683605 (+ 2892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:15.683611 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:15.683797 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:15.683820 (+ 23us) client_negotiation.cc:769] Sending connection context
0811 22:25:15.683871 (+ 51us) client_negotiation.cc:240] Negotiation successful
0811 22:25:15.683964 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17522,"mutex_wait_us":49,"thread_start_us":18782,"threads_started":1}
I20260811 22:25:15.686021 32043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:15.651321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36810 (local address 127.29.106.3:35867)
0811 22:25:15.671816 (+ 20495us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:15.671825 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:15.671888 (+ 63us) server_negotiation.cc:407] Connection header received
0811 22:25:15.672019 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:15.672027 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:15.672262 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:15.672451 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:25:15.674241 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:15.676798 (+ 2557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:15.680878 (+ 4080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:15.683135 (+ 2257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:15.684780 (+ 1645us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:15.685116 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:15.685182 (+ 66us) server_negotiation.cc:299] Negotiation successful
0811 22:25:15.685369 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20155,"thread_start_us":19025,"threads_started":1}
I20260811 22:25:15.687434 31642 tablet_service.cc:1467] Processing CreateTablet for tablet 1e2b90f4c8924fddb581ffb92612ba72 (DEFAULT_TABLE table=test-table [id=ecc781013c3b4be888a500232856bf4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:25:15.688905 31415 tablet_service.cc:1467] Processing CreateTablet for tablet 1e2b90f4c8924fddb581ffb92612ba72 (DEFAULT_TABLE table=test-table [id=ecc781013c3b4be888a500232856bf4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:25:15.689720 31642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e2b90f4c8924fddb581ffb92612ba72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:15.690326 31415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e2b90f4c8924fddb581ffb92612ba72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:15.690619 31898 tablet_service.cc:1467] Processing CreateTablet for tablet 1e2b90f4c8924fddb581ffb92612ba72 (DEFAULT_TABLE table=test-table [id=ecc781013c3b4be888a500232856bf4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:25:15.692490 31898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e2b90f4c8924fddb581ffb92612ba72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:15.709713 32051 tablet_bootstrap.cc:492] T 1e2b90f4c8924fddb581ffb92612ba72 P a40ed35af46f4b048d1ecfe15f291176: Bootstrap starting.
I20260811 22:25:15.712311 32052 tablet_bootstrap.cc:492] T 1e2b90f4c8924fddb581ffb92612ba72 P 636c2e1112304325a775747db1320a92: Bootstrap starting.
I20260811 22:25:15.714156 32054 tablet_bootstrap.cc:492] T 1e2b90f4c8924fddb581ffb92612ba72 P f36008e0e45b432092f555680bfa7104: Bootstrap starting.
I20260811 22:25:15.716567 32051 tablet_bootstrap.cc:654] T 1e2b90f4c8924fddb581ffb92612ba72 P a40ed35af46f4b048d1ecfe15f291176: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:15.718493 32052 tablet_bootstrap.cc:654] T 1e2b90f4c8924fddb581ffb92612ba72 P 636c2e1112304325a775747db1320a92: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:15.719180 32051 log.cc:824] T 1e2b90f4c8924fddb581ffb92612ba72 P a40ed35af46f4b048d1ecfe15f291176: Log is configured to *not* fsync() on all Append() calls
I20260811 22:25:15.719439 32054 tablet_bootstrap.cc:654] T 1e2b90f4c8924fddb581ffb92612ba72 P f36008e0e45b432092f555680bfa7104: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:15.720865 32052 log.cc:824] T 1e2b90f4c8924fddb581ffb92612ba72 P 636c2e1112304325a775747db1320a92: Log is configured to *not* fsync() on all Append() calls
I20260811 22:25:15.721390 32054 log.cc:824] T 1e2b90f4c8924fddb581ffb92612ba72 P f36008e0e45b432092f555680bfa7104: Log is configured to *not* fsync() on all Append() calls
I20260811 22:25:15.725435 32054 tablet_bootstrap.cc:492] T 1e2b90f4c8924fddb581ffb92612ba72 P f36008e0e45b432092f555680bfa7104: No bootstrap required, opened a new log
I20260811 22:25:15.725744 32054 ts_tablet_manager.cc:1397] T 1e2b90f4c8924fddb581ffb92612ba72 P f36008e0e45b432092f555680bfa7104: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.005s
I20260811 22:25:15.730373 32051 tablet_bootstrap.cc:492] T 1e2b90f4c8924fddb581ffb92612ba72 P a40ed35af46f4b048d1ecfe15f291176: No bootstrap required, opened a new log
I20260811 22:25:15.730587 32052 tablet_bootstrap.cc:492] T 1e2b90f4c8924fddb581ffb92612ba72 P 636c2e1112304325a775747db1320a92: No bootstrap required, opened a new log
I20260811 22:25:15.730751 32051 ts_tablet_manager.cc:1397] T 1e2b90f4c8924fddb581ffb92612ba72 P a40ed35af46f4b048d1ecfe15f291176: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.013s
I20260811 22:25:15.730981 32052 ts_tablet_manager.cc:1397] T 1e2b90f4c8924fddb581ffb92612ba72 P 636c2e1112304325a775747db1320a92: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.012s
I20260811 22:25:15.734884 32054 raft_consensus.cc:348] T 1e2b90f4c8924fddb581ffb92612ba72 P f36008e0e45b432092f555680bfa7104 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a40ed35af46f4b048d1ecfe15f291176" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 38321 } } peers { permanent_uuid: "f36008e0e45b432092f555680bfa7104" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 35867 } } peers { permanent_uuid: "636c2e1112304325a775747db1320a92" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 46477 } }
I20260811 22:25:15.735246 32054 raft_consensus.cc:374] T 1e2b90f4c8924fddb581ffb92612ba72 P f36008e0e45b432092f555680bfa7104 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:25:15.735404 32054 raft_consensus.cc:729] T 1e2b90f4c8924fddb581ffb92612ba72 P f36008e0e45b432092f555680bfa7104 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f36008e0e45b432092f555680bfa7104, State: Initialized, Role: FOLLOWER
I20260811 22:25:15.736022 32054 consensus_queue.cc:260] T 1e2b90f4c8924fddb581ffb92612ba72 P f36008e0e45b432092f555680bfa7104 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a40ed35af46f4b048d1ecfe15f291176" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 38321 } } peers { permanent_uuid: "f36008e0e45b432092f555680bfa7104" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 35867 } } peers { permanent_uuid: "636c2e1112304325a775747db1320a92" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 46477 } }
I20260811 22:25:15.740381 32017 heartbeater.cc:503] Master 127.29.106.62:36361 was elected leader, sending a full tablet report...
I20260811 22:25:15.740890 32054 ts_tablet_manager.cc:1428] T 1e2b90f4c8924fddb581ffb92612ba72 P f36008e0e45b432092f555680bfa7104: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260811 22:25:15.741072 32051 raft_consensus.cc:348] T 1e2b90f4c8924fddb581ffb92612ba72 P a40ed35af46f4b048d1ecfe15f291176 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a40ed35af46f4b048d1ecfe15f291176" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 38321 } } peers { permanent_uuid: "f36008e0e45b432092f555680bfa7104" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 35867 } } peers { permanent_uuid: "636c2e1112304325a775747db1320a92" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 46477 } }
I20260811 22:25:15.741510 32051 raft_consensus.cc:374] T 1e2b90f4c8924fddb581ffb92612ba72 P a40ed35af46f4b048d1ecfe15f291176 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:25:15.741623 32051 raft_consensus.cc:729] T 1e2b90f4c8924fddb581ffb92612ba72 P a40ed35af46f4b048d1ecfe15f291176 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a40ed35af46f4b048d1ecfe15f291176, State: Initialized, Role: FOLLOWER
I20260811 22:25:15.741844 32052 raft_consensus.cc:348] T 1e2b90f4c8924fddb581ffb92612ba72 P 636c2e1112304325a775747db1320a92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a40ed35af46f4b048d1ecfe15f291176" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 38321 } } peers { permanent_uuid: "f36008e0e45b432092f555680bfa7104" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 35867 } } peers { permanent_uuid: "636c2e1112304325a775747db1320a92" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 46477 } }
I20260811 22:25:15.742242 32052 raft_consensus.cc:374] T 1e2b90f4c8924fddb581ffb92612ba72 P 636c2e1112304325a775747db1320a92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:25:15.742345 32052 raft_consensus.cc:729] T 1e2b90f4c8924fddb581ffb92612ba72 P 636c2e1112304325a775747db1320a92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 636c2e1112304325a775747db1320a92, State: Initialized, Role: FOLLOWER
I20260811 22:25:15.742264 32051 consensus_queue.cc:260] T 1e2b90f4c8924fddb581ffb92612ba72 P a40ed35af46f4b048d1ecfe15f291176 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a40ed35af46f4b048d1ecfe15f291176" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 38321 } } peers { permanent_uuid: "f36008e0e45b432092f555680bfa7104" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 35867 } } peers { permanent_uuid: "636c2e1112304325a775747db1320a92" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 46477 } }
I20260811 22:25:15.742950 32052 consensus_queue.cc:260] T 1e2b90f4c8924fddb581ffb92612ba72 P 636c2e1112304325a775747db1320a92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a40ed35af46f4b048d1ecfe15f291176" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 38321 } } peers { permanent_uuid: "f36008e0e45b432092f555680bfa7104" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 35867 } } peers { permanent_uuid: "636c2e1112304325a775747db1320a92" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 46477 } }
I20260811 22:25:15.743934 32051 ts_tablet_manager.cc:1428] T 1e2b90f4c8924fddb581ffb92612ba72 P a40ed35af46f4b048d1ecfe15f291176: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260811 22:25:15.744833 31776 heartbeater.cc:503] Master 127.29.106.62:36361 was elected leader, sending a full tablet report...
I20260811 22:25:15.745618 32052 ts_tablet_manager.cc:1428] T 1e2b90f4c8924fddb581ffb92612ba72 P 636c2e1112304325a775747db1320a92: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
W20260811 22:25:15.793272 32018 tablet.cc:2369] T 1e2b90f4c8924fddb581ffb92612ba72 P f36008e0e45b432092f555680bfa7104: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:25:15.825443 32063 raft_consensus.cc:482] T 1e2b90f4c8924fddb581ffb92612ba72 P 636c2e1112304325a775747db1320a92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:25:15.825726 32063 raft_consensus.cc:504] T 1e2b90f4c8924fddb581ffb92612ba72 P 636c2e1112304325a775747db1320a92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a40ed35af46f4b048d1ecfe15f291176" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 38321 } } peers { permanent_uuid: "f36008e0e45b432092f555680bfa7104" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 35867 } } peers { permanent_uuid: "636c2e1112304325a775747db1320a92" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 46477 } }
I20260811 22:25:15.827843 32063 leader_election.cc:288] T 1e2b90f4c8924fddb581ffb92612ba72 P 636c2e1112304325a775747db1320a92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a40ed35af46f4b048d1ecfe15f291176 (127.29.106.1:38321), f36008e0e45b432092f555680bfa7104 (127.29.106.3:35867)
I20260811 22:25:15.834434 32043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:15.828634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:45639 (local address 127.29.106.3:35867)
0811 22:25:15.828817 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:15.828822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:15.829310 (+ 488us) server_negotiation.cc:407] Connection header received
0811 22:25:15.829475 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:15.829479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:15.829536 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:15.829612 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:25:15.830216 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:15.830977 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:15.832714 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:15.833455 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:15.833587 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:15.833661 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:15.834071 (+ 410us) server_negotiation.cc:299] Negotiation successful
0811 22:25:15.834199 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260811 22:25:15.834436 32042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:15.828196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:39205 (local address 127.29.106.1:38321)
0811 22:25:15.828398 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:15.828404 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:15.829052 (+ 648us) server_negotiation.cc:407] Connection header received
0811 22:25:15.829240 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:15.829244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:15.829314 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:15.829394 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:25:15.830039 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:15.830804 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:15.832706 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:15.833491 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:15.833645 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:15.833714 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:15.834086 (+ 372us) server_negotiation.cc:299] Negotiation successful
0811 22:25:15.834209 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260811 22:25:15.834605 32065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:15.828214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.3:35867 (local address 127.29.106.2:45639)
0811 22:25:15.829213 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:15.829227 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:15.829333 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:15.829653 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:15.829657 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:15.829685 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:25:15.830052 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:15.830063 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:15.831116 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:15.831119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:15.832564 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:15.832576 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:15.833677 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:15.833684 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:15.833904 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:15.833949 (+ 45us) client_negotiation.cc:769] Sending connection context
0811 22:25:15.834026 (+ 77us) client_negotiation.cc:240] Negotiation successful
0811 22:25:15.834192 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":912,"mutex_wait_us":57,"thread_start_us":724,"threads_started":1}
I20260811 22:25:15.834637 32064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:15.828061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:38321 (local address 127.29.106.2:39205)
0811 22:25:15.828939 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:15.828955 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:15.829108 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:15.829437 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:15.829442 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:15.829467 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:25:15.829863 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:15.829877 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:15.830937 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:15.830942 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:15.832538 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:15.832550 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:15.833716 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:15.833723 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:15.833913 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:15.833957 (+ 44us) client_negotiation.cc:769] Sending connection context
0811 22:25:15.834044 (+ 87us) client_negotiation.cc:240] Negotiation successful
0811 22:25:15.834205 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"mutex_wait_us":71,"thread_start_us":521,"threads_started":1}
I20260811 22:25:15.835841 31941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2b90f4c8924fddb581ffb92612ba72" candidate_uuid: "636c2e1112304325a775747db1320a92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f36008e0e45b432092f555680bfa7104" is_pre_election: true
I20260811 22:25:15.835840 31444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2b90f4c8924fddb581ffb92612ba72" candidate_uuid: "636c2e1112304325a775747db1320a92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a40ed35af46f4b048d1ecfe15f291176" is_pre_election: true
I20260811 22:25:15.836531 31941 raft_consensus.cc:2454] T 1e2b90f4c8924fddb581ffb92612ba72 P f36008e0e45b432092f555680bfa7104 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 636c2e1112304325a775747db1320a92 in term 0.
I20260811 22:25:15.837265 31571 leader_election.cc:302] T 1e2b90f4c8924fddb581ffb92612ba72 P 636c2e1112304325a775747db1320a92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 636c2e1112304325a775747db1320a92, f36008e0e45b432092f555680bfa7104; no voters:
I20260811 22:25:15.837601 32063 raft_consensus.cc:2781] T 1e2b90f4c8924fddb581ffb92612ba72 P 636c2e1112304325a775747db1320a92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:25:15.837729 32063 raft_consensus.cc:482] T 1e2b90f4c8924fddb581ffb92612ba72 P 636c2e1112304325a775747db1320a92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:25:15.837817 32063 raft_consensus.cc:3037] T 1e2b90f4c8924fddb581ffb92612ba72 P 636c2e1112304325a775747db1320a92 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:15.837721 31444 raft_consensus.cc:2454] T 1e2b90f4c8924fddb581ffb92612ba72 P a40ed35af46f4b048d1ecfe15f291176 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 636c2e1112304325a775747db1320a92 in term 0.
I20260811 22:25:15.840086 32063 raft_consensus.cc:504] T 1e2b90f4c8924fddb581ffb92612ba72 P 636c2e1112304325a775747db1320a92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a40ed35af46f4b048d1ecfe15f291176" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 38321 } } peers { permanent_uuid: "f36008e0e45b432092f555680bfa7104" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 35867 } } peers { permanent_uuid: "636c2e1112304325a775747db1320a92" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 46477 } }
I20260811 22:25:15.840675 32063 leader_election.cc:288] T 1e2b90f4c8924fddb581ffb92612ba72 P 636c2e1112304325a775747db1320a92 [CANDIDATE]: Term 1 election: Requested vote from peers a40ed35af46f4b048d1ecfe15f291176 (127.29.106.1:38321), f36008e0e45b432092f555680bfa7104 (127.29.106.3:35867)
I20260811 22:25:15.841012 31444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2b90f4c8924fddb581ffb92612ba72" candidate_uuid: "636c2e1112304325a775747db1320a92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a40ed35af46f4b048d1ecfe15f291176"
I20260811 22:25:15.841223 31444 raft_consensus.cc:3037] T 1e2b90f4c8924fddb581ffb92612ba72 P a40ed35af46f4b048d1ecfe15f291176 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:15.841213 31941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2b90f4c8924fddb581ffb92612ba72" candidate_uuid: "636c2e1112304325a775747db1320a92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f36008e0e45b432092f555680bfa7104"
I20260811 22:25:15.841396 31941 raft_consensus.cc:3037] T 1e2b90f4c8924fddb581ffb92612ba72 P f36008e0e45b432092f555680bfa7104 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:15.844399 31941 raft_consensus.cc:2454] T 1e2b90f4c8924fddb581ffb92612ba72 P f36008e0e45b432092f555680bfa7104 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 636c2e1112304325a775747db1320a92 in term 1.
I20260811 22:25:15.844399 31444 raft_consensus.cc:2454] T 1e2b90f4c8924fddb581ffb92612ba72 P a40ed35af46f4b048d1ecfe15f291176 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 636c2e1112304325a775747db1320a92 in term 1.
I20260811 22:25:15.844770 31571 leader_election.cc:302] T 1e2b90f4c8924fddb581ffb92612ba72 P 636c2e1112304325a775747db1320a92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 636c2e1112304325a775747db1320a92, f36008e0e45b432092f555680bfa7104; no voters:
I20260811 22:25:15.845049 32063 raft_consensus.cc:2781] T 1e2b90f4c8924fddb581ffb92612ba72 P 636c2e1112304325a775747db1320a92 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:25:15.845486 32063 raft_consensus.cc:686] T 1e2b90f4c8924fddb581ffb92612ba72 P 636c2e1112304325a775747db1320a92 [term 1 LEADER]: Becoming Leader. State: Replica: 636c2e1112304325a775747db1320a92, State: Running, Role: LEADER
I20260811 22:25:15.846019 32063 consensus_queue.cc:237] T 1e2b90f4c8924fddb581ffb92612ba72 P 636c2e1112304325a775747db1320a92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a40ed35af46f4b048d1ecfe15f291176" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 38321 } } peers { permanent_uuid: "f36008e0e45b432092f555680bfa7104" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 35867 } } peers { permanent_uuid: "636c2e1112304325a775747db1320a92" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 46477 } }
I20260811 22:25:15.851848 31189 catalog_manager.cc:5495] T 1e2b90f4c8924fddb581ffb92612ba72 P 636c2e1112304325a775747db1320a92 reported cstate change: term changed from 0 to 1, leader changed from <none> to 636c2e1112304325a775747db1320a92 (127.29.106.2). New cstate: current_term: 1 leader_uuid: "636c2e1112304325a775747db1320a92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a40ed35af46f4b048d1ecfe15f291176" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f36008e0e45b432092f555680bfa7104" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 35867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "636c2e1112304325a775747db1320a92" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 46477 } health_report { overall_health: HEALTHY } } }
Aug 11 22:25:15 dist-test-slave-gjrz krb5kdc[31115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.1@KRBTEST.COM
I20260811 22:25:15.908346 32042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:15.895080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51302 (local address 127.29.106.1:38321)
0811 22:25:15.895237 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:15.895241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:15.895772 (+ 531us) server_negotiation.cc:407] Connection header received
0811 22:25:15.895907 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:15.895912 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:15.895977 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:15.896070 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:15.897520 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:15.898259 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:15.899421 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:15.899797 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:15.899904 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:15.903529 (+ 3625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:15.903550 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:15.903572 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:15.903610 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:15.906081 (+ 2471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:15.906648 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:15.906654 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:15.906656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:15.906783 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:15.907057 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:15.907060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:15.907062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:15.907425 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:15.907639 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:15.907869 (+ 230us) server_negotiation.cc:299] Negotiation successful
0811 22:25:15.908021 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260811 22:25:15.915845 31777 tablet.cc:2369] T 1e2b90f4c8924fddb581ffb92612ba72 P 636c2e1112304325a775747db1320a92: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 22:25:15.925137 31532 tablet.cc:2369] T 1e2b90f4c8924fddb581ffb92612ba72 P a40ed35af46f4b048d1ecfe15f291176: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 11 22:25:15 dist-test-slave-gjrz krb5kdc[31115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487115, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 11 22:25:15 dist-test-slave-gjrz krb5kdc[31115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487115, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.106.1@KRBTEST.COM
I20260811 22:25:15.962220 32042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:15.944712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51306 (local address 127.29.106.1:38321)
0811 22:25:15.944920 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:15.944926 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:15.944949 (+ 23us) server_negotiation.cc:407] Connection header received
0811 22:25:15.945179 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:15.945184 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:15.945247 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:15.945332 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:15.947544 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:15.948310 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:15.950014 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:15.950418 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:15.950540 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:15.955808 (+ 5268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:15.955813 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:15.955816 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:15.955846 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:15.959124 (+ 3278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:15.959733 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:15.959738 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:15.959742 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:15.959938 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:15.960438 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:15.960442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:15.960444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:15.960833 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:15.961309 (+ 476us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:15.961448 (+ 139us) server_negotiation.cc:299] Negotiation successful
0811 22:25:15.961718 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
Aug 11 22:25:15 dist-test-slave-gjrz krb5kdc[31115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 11 22:25:16 dist-test-slave-gjrz krb5kdc[31115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.1@KRBTEST.COM
I20260811 22:25:16.014212 32042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:15.994983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51312 (local address 127.29.106.1:38321)
0811 22:25:15.995190 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:15.995197 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:15.995732 (+ 535us) server_negotiation.cc:407] Connection header received
0811 22:25:15.995914 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:15.995919 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:15.995983 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:15.996068 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:15.998283 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:15.998966 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:16.000929 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:16.001359 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:16.001471 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:16.007367 (+ 5896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:16.007372 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:16.007375 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:16.007409 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:16.010666 (+ 3257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:16.011564 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:16.011568 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:16.011570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:16.011790 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:16.012237 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:16.012241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:16.012243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:16.012693 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:16.013006 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:16.013308 (+ 302us) server_negotiation.cc:299] Negotiation successful
0811 22:25:16.013612 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260811 22:25:16.016702 31378 tablet_service.cc:2908] Rejecting scan request for tablet 1e2b90f4c8924fddb581ffb92612ba72: Uninitialized: clean time has not yet been initialized
I20260811 22:25:16.020418 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 31300
I20260811 22:25:16.035739 32042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:16.021262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51326 (local address 127.29.106.1:38321)
0811 22:25:16.021415 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:16.021419 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:16.021814 (+ 395us) server_negotiation.cc:407] Connection header received
0811 22:25:16.021949 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:16.021952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:16.022013 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:16.022092 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:16.023790 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:16.024430 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:16.025906 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:16.026300 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:16.026387 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:16.029466 (+ 3079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:16.029470 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:16.029473 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:16.029501 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:16.032952 (+ 3451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:16.033706 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:16.033709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:16.033711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:16.033892 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:16.034211 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:16.034214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:16.034216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:16.034656 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:16.034910 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:16.035102 (+ 192us) server_negotiation.cc:299] Negotiation successful
0811 22:25:16.035340 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260811 22:25:16.036563 31523 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:25:16.133986 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 31300
I20260811 22:25:16.158432 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 31538
Aug 11 22:25:16 dist-test-slave-gjrz krb5kdc[31115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.2@KRBTEST.COM
I20260811 22:25:16.173558 32044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:16.159373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41024 (local address 127.29.106.2:46477)
0811 22:25:16.159572 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:16.159577 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:16.159620 (+ 43us) server_negotiation.cc:407] Connection header received
0811 22:25:16.159678 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:16.159682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:16.159744 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:16.159832 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:16.161303 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:16.162109 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:16.163430 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:16.163840 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:16.163937 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:16.167666 (+ 3729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:16.167681 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:16.167698 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:16.167736 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:16.170301 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:16.170922 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:16.170930 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:16.170932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:16.171353 (+ 421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:16.171717 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:16.171720 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:16.171722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:16.172274 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:16.172606 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:16.172792 (+ 186us) server_negotiation.cc:299] Negotiation successful
0811 22:25:16.173123 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260811 22:25:16.174358 31765 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:25:16.264739 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 31538
I20260811 22:25:16.288154 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 31788
Aug 11 22:25:16 dist-test-slave-gjrz krb5kdc[31115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.3@KRBTEST.COM
I20260811 22:25:16.302325 32043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:16.288883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36820 (local address 127.29.106.3:35867)
0811 22:25:16.289104 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:16.289109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:16.289129 (+ 20us) server_negotiation.cc:407] Connection header received
0811 22:25:16.289261 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:16.289265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:16.289328 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:16.289413 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:16.290661 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:16.291449 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:16.292680 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:16.293085 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:16.293198 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:16.297372 (+ 4174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:16.297387 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:16.297405 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:16.297442 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:16.299913 (+ 2471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:16.300529 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:16.300538 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:16.300541 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:16.300677 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:16.301028 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:16.301048 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:16.301050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:16.301387 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:16.301599 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:16.301861 (+ 262us) server_negotiation.cc:299] Negotiation successful
0811 22:25:16.302051 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 22:25:16.303018 32009 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:25:16.389575 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 31788
I20260811 22:25:16.410331 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 31142
I20260811 22:25:16.411588 31247 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:25:16.518226 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 31142
2026-08-11T22:25:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4512 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:25:16 dist-test-slave-gjrz krb5kdc[32108](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:25:16 dist-test-slave-gjrz krb5kdc[32108](info): set up 2 sockets
Aug 11 22:25:16 dist-test-slave-gjrz krb5kdc[32108](info): commencing operation
krb5kdc: starting...
W20260811 22:25:17.779727 30120 mini_kdc.cc:121] Time spent starting KDC: real 1.212s user 0.007s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Aug 11 22:25:17 dist-test-slave-gjrz krb5kdc[32108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:25:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:25:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.106.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
I20260811 22:25:17.946728 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:35421
--webserver_interface=127.29.106.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:38571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:35421
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:25:18.256090 32132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:18.256351 32132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:18.256476 32132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:18.263391 32132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:25:18.263476 32132 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260811 22:25:18.263540 32132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:18.263623 32132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:18.263686 32132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:25:18.263768 32132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:25:18.273017 32132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:38571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:35421
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:35421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.106.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:25:18.274150 32132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:18.275396 32132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:18.286365 32139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:18.286430 32140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:18.286464 32144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:18.287423 32132 server_base.cc:1034] running on GCE node
I20260811 22:25:18.288197 32132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:18.289430 32132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:18.290655 32132 hybrid_clock.cc:648] HybridClock initialized: now 1786487118290602 us; error 36 us; skew 500 ppm
I20260811 22:25:18.290970 32132 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:25:18 dist-test-slave-gjrz krb5kdc[32108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487118, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:18.293958 32132 init.cc:377] Logged in from keytab as kudu/127.29.106.62@KRBTEST.COM (short username kudu)
I20260811 22:25:18.295312 32132 webserver.cc:460] Webserver started at http://127.29.106.62:44945/ using document root <none> and password file <none>
I20260811 22:25:18.295935 32132 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:18.296020 32132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:18.296273 32132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:18.299015 32132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "c2fed88d6bdc4913bd6dd4e7d89e1279"
format_stamp: "Formatted at 2026-08-11 22:25:18 on dist-test-slave-gjrz"
server_key: "55641583cd52b60be0ccfa81f73ac969"
server_key_iv: "1e4f716a0256e4caa6428a29a24f0179"
server_key_version: "encryptionkey@0"
I20260811 22:25:18.299751 32132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "c2fed88d6bdc4913bd6dd4e7d89e1279"
format_stamp: "Formatted at 2026-08-11 22:25:18 on dist-test-slave-gjrz"
server_key: "55641583cd52b60be0ccfa81f73ac969"
server_key_iv: "1e4f716a0256e4caa6428a29a24f0179"
server_key_version: "encryptionkey@0"
I20260811 22:25:18.304632 32132 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260811 22:25:18.307826 32155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:18.309054 32132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 22:25:18.309211 32132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "c2fed88d6bdc4913bd6dd4e7d89e1279"
format_stamp: "Formatted at 2026-08-11 22:25:18 on dist-test-slave-gjrz"
server_key: "55641583cd52b60be0ccfa81f73ac969"
server_key_iv: "1e4f716a0256e4caa6428a29a24f0179"
server_key_version: "encryptionkey@0"
I20260811 22:25:18.309398 32132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:18.325649 32132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:18.335570 32132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:18.335819 32132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:18.356194 32132 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.62:35421
I20260811 22:25:18.356200 32270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.62:35421 every 8 connection(s)
I20260811 22:25:18.357441 32132 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:25:18.361258 32271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:18.361284 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 32132
I20260811 22:25:18.361541 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:25:18.362332 30120 external_mini_cluster.cc:1428] Setting key 7f4e3fa9e7789c21cae6d0abdd10e343
I20260811 22:25:18.374861 32271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2fed88d6bdc4913bd6dd4e7d89e1279: Bootstrap starting.
Aug 11 22:25:18 dist-test-slave-gjrz krb5kdc[32108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:25:18.378280 32271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2fed88d6bdc4913bd6dd4e7d89e1279: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:18.379326 32271 log.cc:824] T 00000000000000000000000000000000 P c2fed88d6bdc4913bd6dd4e7d89e1279: Log is configured to *not* fsync() on all Append() calls
I20260811 22:25:18.382107 32271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2fed88d6bdc4913bd6dd4e7d89e1279: No bootstrap required, opened a new log
I20260811 22:25:18.385124 32277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:18.364339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36258 (local address 127.29.106.62:35421)
0811 22:25:18.365205 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:18.365216 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:18.365255 (+ 39us) server_negotiation.cc:407] Connection header received
0811 22:25:18.367611 (+ 2356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:18.367633 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:18.368062 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:18.368374 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:18.370071 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:18.371367 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:18.373083 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:18.373733 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:18.373863 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:18.378322 (+ 4459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:18.378329 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:18.378342 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:18.378383 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:18.381366 (+ 2983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:18.381998 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:18.382005 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:18.382010 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:18.382190 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:18.382622 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:18.382625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:18.382627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:18.383108 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:18.383366 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:18.383800 (+ 434us) server_negotiation.cc:299] Negotiation successful
0811 22:25:18.384150 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":420,"threads_started":1}
I20260811 22:25:18.389810 32271 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2fed88d6bdc4913bd6dd4e7d89e1279 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2fed88d6bdc4913bd6dd4e7d89e1279" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 35421 } }
I20260811 22:25:18.390128 32271 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2fed88d6bdc4913bd6dd4e7d89e1279 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:25:18.390213 32271 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2fed88d6bdc4913bd6dd4e7d89e1279 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2fed88d6bdc4913bd6dd4e7d89e1279, State: Initialized, Role: FOLLOWER
I20260811 22:25:18.390698 32271 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2fed88d6bdc4913bd6dd4e7d89e1279 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2fed88d6bdc4913bd6dd4e7d89e1279" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 35421 } }
I20260811 22:25:18.390920 32271 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2fed88d6bdc4913bd6dd4e7d89e1279 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:25:18.391040 32271 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2fed88d6bdc4913bd6dd4e7d89e1279 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:25:18.391187 32271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2fed88d6bdc4913bd6dd4e7d89e1279 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:18.393220 32271 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2fed88d6bdc4913bd6dd4e7d89e1279 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2fed88d6bdc4913bd6dd4e7d89e1279" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 35421 } }
I20260811 22:25:18.393667 32271 leader_election.cc:302] T 00000000000000000000000000000000 P c2fed88d6bdc4913bd6dd4e7d89e1279 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2fed88d6bdc4913bd6dd4e7d89e1279; no voters:
I20260811 22:25:18.394199 32271 leader_election.cc:288] T 00000000000000000000000000000000 P c2fed88d6bdc4913bd6dd4e7d89e1279 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:25:18.394294 32279 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2fed88d6bdc4913bd6dd4e7d89e1279 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:25:18.394964 32279 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2fed88d6bdc4913bd6dd4e7d89e1279 [term 1 LEADER]: Becoming Leader. State: Replica: c2fed88d6bdc4913bd6dd4e7d89e1279, State: Running, Role: LEADER
I20260811 22:25:18.395510 32279 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2fed88d6bdc4913bd6dd4e7d89e1279 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2fed88d6bdc4913bd6dd4e7d89e1279" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 35421 } }
I20260811 22:25:18.396082 32271 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2fed88d6bdc4913bd6dd4e7d89e1279 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:25:18.398077 32282 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2fed88d6bdc4913bd6dd4e7d89e1279 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2fed88d6bdc4913bd6dd4e7d89e1279" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2fed88d6bdc4913bd6dd4e7d89e1279" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 35421 } } }
I20260811 22:25:18.398581 32282 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2fed88d6bdc4913bd6dd4e7d89e1279 [sys.catalog]: This master's current role is: LEADER
I20260811 22:25:18.398161 32285 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2fed88d6bdc4913bd6dd4e7d89e1279 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2fed88d6bdc4913bd6dd4e7d89e1279. Latest consensus state: current_term: 1 leader_uuid: "c2fed88d6bdc4913bd6dd4e7d89e1279" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2fed88d6bdc4913bd6dd4e7d89e1279" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 35421 } } }
I20260811 22:25:18.399089 32285 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2fed88d6bdc4913bd6dd4e7d89e1279 [sys.catalog]: This master's current role is: LEADER
I20260811 22:25:18.404070 32291 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:25:18.412149 32291 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:25:18.421043 32291 catalog_manager.cc:1347] Generated new cluster ID: 53028ad613124f1080b80c0c2c14f089
I20260811 22:25:18.421142 32291 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:25:18.435084 32291 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:25:18.436250 32291 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:25:18.442500 32291 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2fed88d6bdc4913bd6dd4e7d89e1279: Generated new TSK 0
I20260811 22:25:18.443179 32291 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.106.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab.
Entry for principal kudu/127.29.106.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab.
I20260811 22:25:18.492460 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.1:0
--local_ip_for_outbound_sockets=127.29.106.1
--webserver_interface=127.29.106.1
--webserver_port=0
--tserver_master_addrs=127.29.106.62:35421
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab
--principal=kudu/127.29.106.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:38571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:25:18.829861 32315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:18.830101 32315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:18.830192 32315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:18.836992 32315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:18.837082 32315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:18.837160 32315 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 22:25:18.837237 32315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.1
I20260811 22:25:18.846271 32315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:38571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab
--principal=kudu/127.29.106.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.106.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.106.62:35421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.1
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:25:18.847343 32315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:18.848529 32315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:18.860435 32320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:18.860486 32332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:18.860517 32327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:18.861243 32315 server_base.cc:1034] running on GCE node
I20260811 22:25:18.861776 32315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:18.862584 32315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:18.863843 32315 hybrid_clock.cc:648] HybridClock initialized: now 1786487118863766 us; error 37 us; skew 500 ppm
I20260811 22:25:18.864130 32315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:25:18 dist-test-slave-gjrz krb5kdc[32108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487118, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:18.867009 32315 init.cc:377] Logged in from keytab as kudu/127.29.106.1@KRBTEST.COM (short username kudu)
I20260811 22:25:18.868417 32315 webserver.cc:460] Webserver started at http://127.29.106.1:35047/ using document root <none> and password file <none>
I20260811 22:25:18.869045 32315 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:18.869144 32315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:18.869426 32315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:18.872241 32315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-0/data/instance:
uuid: "9664a73064234bc4912c8ddb99a59cb0"
format_stamp: "Formatted at 2026-08-11 22:25:18 on dist-test-slave-gjrz"
server_key: "9b102cfb35f7fd095d31e4de71d5a9d2"
server_key_iv: "e304a49b12e46b5f4e2773d135e11edb"
server_key_version: "encryptionkey@0"
I20260811 22:25:18.872998 32315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance:
uuid: "9664a73064234bc4912c8ddb99a59cb0"
format_stamp: "Formatted at 2026-08-11 22:25:18 on dist-test-slave-gjrz"
server_key: "9b102cfb35f7fd095d31e4de71d5a9d2"
server_key_iv: "e304a49b12e46b5f4e2773d135e11edb"
server_key_version: "encryptionkey@0"
I20260811 22:25:18.877827 32315 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 22:25:18.881207 32342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:18.882486 32315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 22:25:18.882654 32315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-0/wal
uuid: "9664a73064234bc4912c8ddb99a59cb0"
format_stamp: "Formatted at 2026-08-11 22:25:18 on dist-test-slave-gjrz"
server_key: "9b102cfb35f7fd095d31e4de71d5a9d2"
server_key_iv: "e304a49b12e46b5f4e2773d135e11edb"
server_key_version: "encryptionkey@0"
I20260811 22:25:18.882841 32315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:18.899299 32315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:18.903041 32315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:18.903304 32315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:18.906049 32315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:25:18.906154 32315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:18.906368 32315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:25:18.906493 32315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 22:25:18 dist-test-slave-gjrz krb5kdc[32108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487118, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.1@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:25:18.927572 32353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:18.906353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:43065 (local address 127.29.106.62:35421)
0811 22:25:18.906784 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:18.906791 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:18.908051 (+ 1260us) server_negotiation.cc:407] Connection header received
0811 22:25:18.910110 (+ 2059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:18.910116 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:18.910197 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:18.910291 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:18.912401 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:18.913553 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:18.916394 (+ 2841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:18.916932 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:18.917061 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:18.921831 (+ 4770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:18.921836 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:18.921839 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:18.921881 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:18.924299 (+ 2418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:18.925194 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:18.925197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:18.925200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:18.925317 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:18.925790 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:18.925793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:18.925795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:18.926077 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:18.926192 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:18.926976 (+ 784us) server_negotiation.cc:299] Negotiation successful
0811 22:25:18.927159 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":223,"threads_started":1}
I20260811 22:25:18.929165 32357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:18.907028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:35421 (local address 127.29.106.1:43065)
0811 22:25:18.907877 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:18.907912 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:18.909845 (+ 1933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:18.910570 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:18.910581 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:18.911338 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:18.912143 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:18.912166 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:18.913792 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:18.913798 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:18.916216 (+ 2418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:18.916239 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:18.917370 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:18.917378 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:18.917681 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:18.918411 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:18.918432 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:18.921599 (+ 3167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:18.924444 (+ 2845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:18.924455 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:18.924471 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:18.925073 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:18.925412 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:18.925417 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:18.925421 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:18.925677 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:18.926183 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:18.926192 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:18.926620 (+ 428us) client_negotiation.cc:769] Sending connection context
0811 22:25:18.926927 (+ 307us) client_negotiation.cc:240] Negotiation successful
0811 22:25:18.927510 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":390,"threads_started":1}
I20260811 22:25:18.948961 32459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:18.935403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:35421 (local address 127.29.106.1:39683)
0811 22:25:18.936290 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:18.936315 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:18.936470 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:18.936942 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:18.936946 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:18.937538 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:18.937868 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:18.937881 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:18.938820 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:18.938825 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:18.940049 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:18.940062 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:18.940772 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:18.940779 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:18.940967 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:18.941578 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:18.941601 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:18.942562 (+ 961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:18.946241 (+ 3679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:18.946250 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:18.946253 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:18.946716 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:18.947129 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:18.947132 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:18.947134 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:18.947303 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:18.947926 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:18.947933 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:18.948221 (+ 288us) client_negotiation.cc:769] Sending connection context
0811 22:25:18.948387 (+ 166us) client_negotiation.cc:240] Negotiation successful
0811 22:25:18.948660 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"thread_start_us":546,"threads_started":1}
I20260811 22:25:18.949105 32353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:18.935450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:39683 (local address 127.29.106.62:35421)
0811 22:25:18.935728 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:18.935734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:18.936407 (+ 673us) server_negotiation.cc:407] Connection header received
0811 22:25:18.936613 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:18.936619 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:18.936694 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:18.936788 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:18.938021 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:18.938696 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:18.940212 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:18.940595 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:18.940724 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:18.942788 (+ 2064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:18.942795 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:18.942801 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:18.942845 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:18.946111 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:18.946841 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:18.946847 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:18.946850 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:18.947008 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:18.947436 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:18.947442 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:18.947445 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:18.947774 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:18.948174 (+ 400us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:18.948529 (+ 355us) server_negotiation.cc:299] Negotiation successful
0811 22:25:18.948765 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20260811 22:25:18.959486 32315 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.1:43297
I20260811 22:25:18.959508 32565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.1:43297 every 8 connection(s)
I20260811 22:25:18.960623 32315 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
I20260811 22:25:18.961414 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 32315
I20260811 22:25:18.961642 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance
I20260811 22:25:18.962308 30120 external_mini_cluster.cc:1428] Setting key b13a06d11fddd723771bcef45bff83f8
I20260811 22:25:18.962581 32568 heartbeater.cc:347] Connected to a master server at 127.29.106.62:35421
I20260811 22:25:18.962785 32568 heartbeater.cc:464] Registering TS with master...
I20260811 22:25:18.963569 32568 heartbeater.cc:511] Master 127.29.106.62:35421 requested a full tablet report, sending...
I20260811 22:25:18.965478 32185 ts_manager.cc:194] Registered new tserver with Master: 9664a73064234bc4912c8ddb99a59cb0 (127.29.106.1:43297)
I20260811 22:25:18.967134 32185 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.1@KRBTEST.COM'} at 127.29.106.1:43065
WARNING: no policy specified for kudu/127.29.106.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab.
Entry for principal kudu/127.29.106.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab.
I20260811 22:25:19.000550 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.2:0
--local_ip_for_outbound_sockets=127.29.106.2
--webserver_interface=127.29.106.2
--webserver_port=0
--tserver_master_addrs=127.29.106.62:35421
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab
--principal=kudu/127.29.106.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:38571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:25:19.313331 32574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:19.313598 32574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:19.313728 32574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:19.321053 32574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:19.321154 32574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:19.321271 32574 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 22:25:19.321375 32574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.2
I20260811 22:25:19.330708 32574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:38571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab
--principal=kudu/127.29.106.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.106.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.106.62:35421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.2
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:25:19.331984 32574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:19.333323 32574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:19.346745 32581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:19.346752 32586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:19.346796 32583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:19.347648 32574 server_base.cc:1034] running on GCE node
I20260811 22:25:19.348174 32574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:19.348958 32574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:19.350175 32574 hybrid_clock.cc:648] HybridClock initialized: now 1786487119350107 us; error 42 us; skew 500 ppm
I20260811 22:25:19.350445 32574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:25:19 dist-test-slave-gjrz krb5kdc[32108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487119, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:19.353375 32574 init.cc:377] Logged in from keytab as kudu/127.29.106.2@KRBTEST.COM (short username kudu)
I20260811 22:25:19.357124 32574 webserver.cc:460] Webserver started at http://127.29.106.2:33447/ using document root <none> and password file <none>
I20260811 22:25:19.357748 32574 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:19.357848 32574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:19.358143 32574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:19.360909 32574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-1/data/instance:
uuid: "85f7acc1fba04ddebccc688885357742"
format_stamp: "Formatted at 2026-08-11 22:25:19 on dist-test-slave-gjrz"
server_key: "ff07ed6e7fef6daadbdeb6b350fca672"
server_key_iv: "e5a67316b7a44181f696e7480a28840b"
server_key_version: "encryptionkey@0"
I20260811 22:25:19.361769 32574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance:
uuid: "85f7acc1fba04ddebccc688885357742"
format_stamp: "Formatted at 2026-08-11 22:25:19 on dist-test-slave-gjrz"
server_key: "ff07ed6e7fef6daadbdeb6b350fca672"
server_key_iv: "e5a67316b7a44181f696e7480a28840b"
server_key_version: "encryptionkey@0"
I20260811 22:25:19.366824 32574 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260811 22:25:19.370826 32598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:19.372143 32574 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:25:19.372262 32574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-1/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-1/wal
uuid: "85f7acc1fba04ddebccc688885357742"
format_stamp: "Formatted at 2026-08-11 22:25:19 on dist-test-slave-gjrz"
server_key: "ff07ed6e7fef6daadbdeb6b350fca672"
server_key_iv: "e5a67316b7a44181f696e7480a28840b"
server_key_version: "encryptionkey@0"
I20260811 22:25:19.372421 32574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:19.384769 32574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:19.388669 32574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:19.388880 32574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:19.391214 32574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:25:19.391335 32574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:19.391471 32574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:25:19.391536 32574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 22:25:19 dist-test-slave-gjrz krb5kdc[32108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487119, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.2@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:25:19.412844 32353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:19.391899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:57093 (local address 127.29.106.62:35421)
0811 22:25:19.392101 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:19.392107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:19.393337 (+ 1230us) server_negotiation.cc:407] Connection header received
0811 22:25:19.395445 (+ 2108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:19.395452 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:19.395545 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:19.395675 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:19.397675 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:19.398662 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:19.401133 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:19.401717 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:19.401826 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:19.406222 (+ 4396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:19.406228 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:19.406233 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:19.406277 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:19.409207 (+ 2930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:19.410092 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:19.410097 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:19.410101 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:19.410265 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:19.410801 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:19.410807 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:19.410810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:19.411136 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:19.411306 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:19.412369 (+ 1063us) server_negotiation.cc:299] Negotiation successful
0811 22:25:19.412602 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260811 22:25:19.414495 32616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:19.392356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:35421 (local address 127.29.106.2:57093)
0811 22:25:19.393171 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:19.393214 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:19.395151 (+ 1937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:19.395892 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:19.395902 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:19.396558 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:19.397434 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:19.397464 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:19.398837 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:19.398841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:19.400994 (+ 2153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:19.401010 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:19.401973 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:19.401985 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:19.402229 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:19.402979 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:19.403017 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:19.406006 (+ 2989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:19.409340 (+ 3334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:19.409350 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:19.409365 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:19.409936 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:19.410408 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:19.410422 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:19.410425 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:19.410639 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:19.411373 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:19.411381 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:19.411867 (+ 486us) client_negotiation.cc:769] Sending connection context
0811 22:25:19.412239 (+ 372us) client_negotiation.cc:240] Negotiation successful
0811 22:25:19.412782 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":324,"threads_started":1}
I20260811 22:25:19.429672 32574 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.2:44367
I20260811 22:25:19.429700 348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.2:44367 every 8 connection(s)
I20260811 22:25:19.430874 32574 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
I20260811 22:25:19.433355 349 heartbeater.cc:347] Connected to a master server at 127.29.106.62:35421
I20260811 22:25:19.433580 349 heartbeater.cc:464] Registering TS with master...
I20260811 22:25:19.434347 349 heartbeater.cc:511] Master 127.29.106.62:35421 requested a full tablet report, sending...
I20260811 22:25:19.434871 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 32574
I20260811 22:25:19.435081 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance
I20260811 22:25:19.435580 30120 external_mini_cluster.cc:1428] Setting key d52dc74455c54780f1f49c997ad68c58
I20260811 22:25:19.435865 32185 ts_manager.cc:194] Registered new tserver with Master: 85f7acc1fba04ddebccc688885357742 (127.29.106.2:44367)
I20260811 22:25:19.436939 32185 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.2@KRBTEST.COM'} at 127.29.106.2:57093
I20260811 22:25:19.439406 32353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:19.425628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:35609 (local address 127.29.106.62:35421)
0811 22:25:19.425798 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:19.425805 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:19.426604 (+ 799us) server_negotiation.cc:407] Connection header received
0811 22:25:19.426862 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:19.426867 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:19.426957 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:19.427057 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:19.428262 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:19.429074 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:19.430532 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:19.430965 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:19.431071 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:19.433105 (+ 2034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:19.433111 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:19.433116 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:19.433152 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:19.436467 (+ 3315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:19.437372 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:19.437376 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:19.437379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:19.437555 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:19.437968 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:19.437973 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:19.437977 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:19.438308 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:19.438455 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:19.438849 (+ 394us) server_negotiation.cc:299] Negotiation successful
0811 22:25:19.439052 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260811 22:25:19.439700 332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:19.425737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:35421 (local address 127.29.106.2:35609)
0811 22:25:19.426497 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:19.426518 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:19.426670 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:19.427215 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:19.427218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:19.427825 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:19.428125 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:19.428134 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:19.429208 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:19.429214 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:19.430388 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:19.430402 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:19.431144 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:19.431151 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:19.431392 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:19.432105 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:19.432125 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:19.432929 (+ 804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:19.436594 (+ 3665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:19.436600 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:19.436604 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:19.437128 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:19.437682 (+ 554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:19.437686 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:19.437689 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:19.437850 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:19.438434 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:19.438439 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:19.438617 (+ 178us) client_negotiation.cc:769] Sending connection context
0811 22:25:19.438734 (+ 117us) client_negotiation.cc:240] Negotiation successful
0811 22:25:19.439033 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":413,"threads_started":1}
WARNING: no policy specified for kudu/127.29.106.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab.
Entry for principal kudu/127.29.106.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab.
I20260811 22:25:19.474363 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.3:0
--local_ip_for_outbound_sockets=127.29.106.3
--webserver_interface=127.29.106.3
--webserver_port=0
--tserver_master_addrs=127.29.106.62:35421
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab
--principal=kudu/127.29.106.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:38571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:25:19.836127 355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:19.836376 355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:19.836510 355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:19.843465 355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:19.843570 355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:19.843689 355 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 22:25:19.843796 355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.3
I20260811 22:25:19.852751 355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:38571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab
--principal=kudu/127.29.106.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.106.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.106.62:35421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.3
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:25:19.853816 355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:19.855336 355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:19.868871 367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:19.868966 362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:19.868947 363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:19.872552 355 server_base.cc:1034] running on GCE node
I20260811 22:25:19.873068 355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:19.873942 355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:19.875164 355 hybrid_clock.cc:648] HybridClock initialized: now 1786487119875087 us; error 47 us; skew 500 ppm
I20260811 22:25:19.875478 355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:25:19 dist-test-slave-gjrz krb5kdc[32108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487119, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:19.878357 355 init.cc:377] Logged in from keytab as kudu/127.29.106.3@KRBTEST.COM (short username kudu)
I20260811 22:25:19.879639 355 webserver.cc:460] Webserver started at http://127.29.106.3:44111/ using document root <none> and password file <none>
I20260811 22:25:19.880254 355 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:19.880332 355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:19.880590 355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:19.883246 355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-2/data/instance:
uuid: "8b139b7de16d4c39acd5ba4d9ab8648d"
format_stamp: "Formatted at 2026-08-11 22:25:19 on dist-test-slave-gjrz"
server_key: "652210bf4511b583489e1445dbc3d917"
server_key_iv: "da2544bd7dbba7e356e499665f374e73"
server_key_version: "encryptionkey@0"
I20260811 22:25:19.884038 355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance:
uuid: "8b139b7de16d4c39acd5ba4d9ab8648d"
format_stamp: "Formatted at 2026-08-11 22:25:19 on dist-test-slave-gjrz"
server_key: "652210bf4511b583489e1445dbc3d917"
server_key_iv: "da2544bd7dbba7e356e499665f374e73"
server_key_version: "encryptionkey@0"
I20260811 22:25:19.888767 355 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 22:25:19.891875 383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:19.893039 355 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:25:19.893187 355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-2/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-2/wal
uuid: "8b139b7de16d4c39acd5ba4d9ab8648d"
format_stamp: "Formatted at 2026-08-11 22:25:19 on dist-test-slave-gjrz"
server_key: "652210bf4511b583489e1445dbc3d917"
server_key_iv: "da2544bd7dbba7e356e499665f374e73"
server_key_version: "encryptionkey@0"
I20260811 22:25:19.893369 355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:19.915179 355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:19.919365 355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:19.919601 355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:19.922097 355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:25:19.922199 355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:19.922338 355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:25:19.922415 355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 22:25:19 dist-test-slave-gjrz krb5kdc[32108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487119, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.3@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:25:19.944512 32353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:19.922633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:46375 (local address 127.29.106.62:35421)
0811 22:25:19.922827 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:19.922834 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:19.924411 (+ 1577us) server_negotiation.cc:407] Connection header received
0811 22:25:19.926331 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:19.926336 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:19.926432 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:19.926547 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:19.928811 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:19.929718 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:19.932621 (+ 2903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:19.933017 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:19.933097 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:19.937699 (+ 4602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:19.937706 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:19.937710 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:19.937751 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:19.940765 (+ 3014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:19.941665 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:19.941671 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:19.941675 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:19.941805 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:19.942660 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:19.942665 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:19.942668 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:19.942997 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:19.943135 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:19.944083 (+ 948us) server_negotiation.cc:299] Negotiation successful
0811 22:25:19.944262 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260811 22:25:19.946010 393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:19.923461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:35421 (local address 127.29.106.3:46375)
0811 22:25:19.924263 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:19.924306 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:19.926067 (+ 1761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:19.926833 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:19.926844 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:19.927680 (+ 836us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:19.928610 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:19.928639 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:19.929900 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:19.929906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:19.932420 (+ 2514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:19.932437 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:19.933512 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:19.933523 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:19.933781 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:19.934466 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:19.934502 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:19.937433 (+ 2931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:19.940927 (+ 3494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:19.940935 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:19.940950 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:19.941527 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:19.942238 (+ 711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:19.942243 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:19.942246 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:19.942491 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:19.943233 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:19.943244 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:19.943639 (+ 395us) client_negotiation.cc:769] Sending connection context
0811 22:25:19.944711 (+ 1072us) client_negotiation.cc:240] Negotiation successful
0811 22:25:19.945096 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":325,"threads_started":1}
I20260811 22:25:19.963780 32353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:19.950921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:54665 (local address 127.29.106.62:35421)
0811 22:25:19.951078 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:19.951083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:19.951931 (+ 848us) server_negotiation.cc:407] Connection header received
0811 22:25:19.952066 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:19.952069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:19.952122 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:19.952202 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:19.953503 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:19.954402 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:19.956308 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:19.956704 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:19.956805 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:19.958920 (+ 2115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:19.958926 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:19.958931 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:19.958962 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:19.961351 (+ 2389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:19.961936 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:19.961941 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:19.961945 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:19.962070 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:19.962387 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:19.962392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:19.962396 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:19.962713 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:19.962839 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:19.963218 (+ 379us) server_negotiation.cc:299] Negotiation successful
0811 22:25:19.963459 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 22:25:19.963869 519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:19.950942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:35421 (local address 127.29.106.3:54665)
0811 22:25:19.951825 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:19.951845 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:19.951976 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:19.952250 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:19.952255 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:19.952896 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:19.953372 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:19.953389 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:19.954569 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:19.954573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:19.956182 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:19.956194 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:19.956886 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:19.956892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:19.957046 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:19.957652 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:19.957668 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:19.958686 (+ 1018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:19.961466 (+ 2780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:19.961472 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:19.961476 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:19.961821 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:19.962167 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:19.962172 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:19.962176 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:19.962291 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:19.962820 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:19.962825 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:19.962986 (+ 161us) client_negotiation.cc:769] Sending connection context
0811 22:25:19.963102 (+ 116us) client_negotiation.cc:240] Negotiation successful
0811 22:25:19.963414 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":563,"threads_started":1}
I20260811 22:25:19.970032 32568 heartbeater.cc:503] Master 127.29.106.62:35421 was elected leader, sending a full tablet report...
I20260811 22:25:19.974211 355 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.3:36999
I20260811 22:25:19.974233 592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.3:36999 every 8 connection(s)
I20260811 22:25:19.975255 355 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
I20260811 22:25:19.977078 593 heartbeater.cc:347] Connected to a master server at 127.29.106.62:35421
I20260811 22:25:19.977273 593 heartbeater.cc:464] Registering TS with master...
I20260811 22:25:19.977815 593 heartbeater.cc:511] Master 127.29.106.62:35421 requested a full tablet report, sending...
I20260811 22:25:19.978941 32183 ts_manager.cc:194] Registered new tserver with Master: 8b139b7de16d4c39acd5ba4d9ab8648d (127.29.106.3:36999)
I20260811 22:25:19.979784 32183 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.3@KRBTEST.COM'} at 127.29.106.3:46375
I20260811 22:25:19.983768 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 355
I20260811 22:25:19.983934 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance
I20260811 22:25:19.984330 30120 external_mini_cluster.cc:1428] Setting key 4f083a956f3b9fa962b43e6ff1e9f33d
I20260811 22:25:19.986931 30120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 11 22:25:20 dist-test-slave-gjrz krb5kdc[32108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487120, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 11 22:25:20 dist-test-slave-gjrz krb5kdc[32108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487120, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:25:20.031342 32353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:20.018455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41556 (local address 127.29.106.62:35421)
0811 22:25:20.018674 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:20.018680 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:20.018915 (+ 235us) server_negotiation.cc:407] Connection header received
0811 22:25:20.019095 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:20.019099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:20.019176 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:20.019287 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:20.020600 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.021326 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.022582 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.022978 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.023084 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:20.026560 (+ 3476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:20.026565 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:20.026568 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:20.026606 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:20.028937 (+ 2331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:20.029505 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:20.029510 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:20.029512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:20.029636 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:20.029986 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:20.029990 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:20.029994 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:20.030259 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:20.030485 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:20.030779 (+ 294us) server_negotiation.cc:299] Negotiation successful
0811 22:25:20.031000 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260811 22:25:20.034904 32183 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41556:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:25:20.041451 32183 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:25:20.064283 620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:20.053456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51108 (local address 127.29.106.1:43297)
0811 22:25:20.055397 (+ 1941us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:20.055406 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:20.055434 (+ 28us) server_negotiation.cc:407] Connection header received
0811 22:25:20.055541 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:20.055550 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:20.055794 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:20.055977 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:25:20.056868 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.058241 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.060885 (+ 2644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.062357 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.062478 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:20.062689 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:20.062919 (+ 230us) server_negotiation.cc:299] Negotiation successful
0811 22:25:20.063075 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1757,"thread_start_us":1661,"threads_started":1}
I20260811 22:25:20.064478 621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:20.052899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:43297 (local address 127.0.0.1:51108)
0811 22:25:20.055005 (+ 2106us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:20.055028 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:20.055147 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:20.056222 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:20.056229 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:20.056255 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:25:20.056661 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:20.056673 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.058377 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.058385 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:20.060717 (+ 2332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:20.060732 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.062538 (+ 1806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.062545 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:20.062724 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:20.062762 (+ 38us) client_negotiation.cc:769] Sending connection context
0811 22:25:20.062865 (+ 103us) client_negotiation.cc:240] Negotiation successful
0811 22:25:20.063046 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2001,"thread_start_us":2477,"threads_started":1}
I20260811 22:25:20.064504 615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:20.052396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.2:44367 (local address 127.0.0.1:40674)
0811 22:25:20.054099 (+ 1703us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:20.054143 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:20.054293 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:20.057692 (+ 3399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:20.057697 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:20.057721 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:25:20.058093 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:20.058108 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.059897 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.059903 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:20.061643 (+ 1740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:20.061659 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.063140 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.063147 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:20.063321 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:20.063355 (+ 34us) client_negotiation.cc:769] Sending connection context
0811 22:25:20.063421 (+ 66us) client_negotiation.cc:240] Negotiation successful
0811 22:25:20.063544 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1465,"thread_start_us":1316,"threads_started":1}
I20260811 22:25:20.065045 619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:20.054460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.3:36999 (local address 127.0.0.1:39716)
0811 22:25:20.056576 (+ 2116us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:20.056603 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:20.056719 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:20.058100 (+ 1381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:20.058105 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:20.058129 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:25:20.058525 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:20.058539 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.060260 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.060264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:20.062210 (+ 1946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:20.062225 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.064308 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.064315 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:20.064494 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:20.064524 (+ 30us) client_negotiation.cc:769] Sending connection context
0811 22:25:20.064622 (+ 98us) client_negotiation.cc:240] Negotiation successful
0811 22:25:20.064742 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1037,"thread_start_us":309,"threads_started":1}
I20260811 22:25:20.065274 618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:20.052629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40674 (local address 127.29.106.2:44367)
0811 22:25:20.057183 (+ 4554us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:20.057191 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:20.057217 (+ 26us) server_negotiation.cc:407] Connection header received
0811 22:25:20.057304 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:20.057311 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:20.057514 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:20.057688 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:25:20.058301 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.059746 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.061835 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.062909 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.063650 (+ 741us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:20.063843 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:20.063910 (+ 67us) server_negotiation.cc:299] Negotiation successful
0811 22:25:20.064061 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4400,"thread_start_us":4270,"threads_started":1}
I20260811 22:25:20.065822 623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:20.054877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39716 (local address 127.29.106.3:36999)
0811 22:25:20.057551 (+ 2674us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:20.057557 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:20.057582 (+ 25us) server_negotiation.cc:407] Connection header received
0811 22:25:20.057669 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:20.057675 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:20.057883 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:20.058071 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:25:20.058723 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.060128 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.062381 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.063840 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.063955 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:20.064157 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:20.065356 (+ 1199us) server_negotiation.cc:299] Negotiation successful
0811 22:25:20.065517 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2551,"thread_start_us":2371,"threads_started":1}
I20260811 22:25:20.068171 32428 tablet_service.cc:1467] Processing CreateTablet for tablet 2c2f3efbcd984f57b698448fdf2caa65 (DEFAULT_TABLE table=test-table [id=45f361ad02ba4525ab8449020f08aad0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:25:20.068403 32686 tablet_service.cc:1467] Processing CreateTablet for tablet 2c2f3efbcd984f57b698448fdf2caa65 (DEFAULT_TABLE table=test-table [id=45f361ad02ba4525ab8449020f08aad0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:25:20.069468 32428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c2f3efbcd984f57b698448fdf2caa65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:20.069190 463 tablet_service.cc:1467] Processing CreateTablet for tablet 2c2f3efbcd984f57b698448fdf2caa65 (DEFAULT_TABLE table=test-table [id=45f361ad02ba4525ab8449020f08aad0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:25:20.069701 32686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c2f3efbcd984f57b698448fdf2caa65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:20.070462 463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c2f3efbcd984f57b698448fdf2caa65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:20.079932 625 tablet_bootstrap.cc:492] T 2c2f3efbcd984f57b698448fdf2caa65 P 9664a73064234bc4912c8ddb99a59cb0: Bootstrap starting.
I20260811 22:25:20.083551 627 tablet_bootstrap.cc:492] T 2c2f3efbcd984f57b698448fdf2caa65 P 85f7acc1fba04ddebccc688885357742: Bootstrap starting.
I20260811 22:25:20.083850 625 tablet_bootstrap.cc:654] T 2c2f3efbcd984f57b698448fdf2caa65 P 9664a73064234bc4912c8ddb99a59cb0: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:20.085006 629 tablet_bootstrap.cc:492] T 2c2f3efbcd984f57b698448fdf2caa65 P 8b139b7de16d4c39acd5ba4d9ab8648d: Bootstrap starting.
I20260811 22:25:20.085141 625 log.cc:824] T 2c2f3efbcd984f57b698448fdf2caa65 P 9664a73064234bc4912c8ddb99a59cb0: Log is configured to *not* fsync() on all Append() calls
I20260811 22:25:20.087164 627 tablet_bootstrap.cc:654] T 2c2f3efbcd984f57b698448fdf2caa65 P 85f7acc1fba04ddebccc688885357742: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:20.088043 629 tablet_bootstrap.cc:654] T 2c2f3efbcd984f57b698448fdf2caa65 P 8b139b7de16d4c39acd5ba4d9ab8648d: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:20.088380 625 tablet_bootstrap.cc:492] T 2c2f3efbcd984f57b698448fdf2caa65 P 9664a73064234bc4912c8ddb99a59cb0: No bootstrap required, opened a new log
I20260811 22:25:20.088392 627 log.cc:824] T 2c2f3efbcd984f57b698448fdf2caa65 P 85f7acc1fba04ddebccc688885357742: Log is configured to *not* fsync() on all Append() calls
I20260811 22:25:20.088675 625 ts_tablet_manager.cc:1397] T 2c2f3efbcd984f57b698448fdf2caa65 P 9664a73064234bc4912c8ddb99a59cb0: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260811 22:25:20.088985 629 log.cc:824] T 2c2f3efbcd984f57b698448fdf2caa65 P 8b139b7de16d4c39acd5ba4d9ab8648d: Log is configured to *not* fsync() on all Append() calls
I20260811 22:25:20.097956 629 tablet_bootstrap.cc:492] T 2c2f3efbcd984f57b698448fdf2caa65 P 8b139b7de16d4c39acd5ba4d9ab8648d: No bootstrap required, opened a new log
I20260811 22:25:20.098274 629 ts_tablet_manager.cc:1397] T 2c2f3efbcd984f57b698448fdf2caa65 P 8b139b7de16d4c39acd5ba4d9ab8648d: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.008s
I20260811 22:25:20.098542 627 tablet_bootstrap.cc:492] T 2c2f3efbcd984f57b698448fdf2caa65 P 85f7acc1fba04ddebccc688885357742: No bootstrap required, opened a new log
I20260811 22:25:20.099020 627 ts_tablet_manager.cc:1397] T 2c2f3efbcd984f57b698448fdf2caa65 P 85f7acc1fba04ddebccc688885357742: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.004s
I20260811 22:25:20.102919 625 raft_consensus.cc:348] T 2c2f3efbcd984f57b698448fdf2caa65 P 9664a73064234bc4912c8ddb99a59cb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f7acc1fba04ddebccc688885357742" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44367 } } peers { permanent_uuid: "8b139b7de16d4c39acd5ba4d9ab8648d" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36999 } } peers { permanent_uuid: "9664a73064234bc4912c8ddb99a59cb0" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43297 } }
I20260811 22:25:20.103804 625 raft_consensus.cc:374] T 2c2f3efbcd984f57b698448fdf2caa65 P 9664a73064234bc4912c8ddb99a59cb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:25:20.104082 625 raft_consensus.cc:729] T 2c2f3efbcd984f57b698448fdf2caa65 P 9664a73064234bc4912c8ddb99a59cb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9664a73064234bc4912c8ddb99a59cb0, State: Initialized, Role: FOLLOWER
I20260811 22:25:20.105252 625 consensus_queue.cc:260] T 2c2f3efbcd984f57b698448fdf2caa65 P 9664a73064234bc4912c8ddb99a59cb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f7acc1fba04ddebccc688885357742" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44367 } } peers { permanent_uuid: "8b139b7de16d4c39acd5ba4d9ab8648d" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36999 } } peers { permanent_uuid: "9664a73064234bc4912c8ddb99a59cb0" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43297 } }
I20260811 22:25:20.109598 629 raft_consensus.cc:348] T 2c2f3efbcd984f57b698448fdf2caa65 P 8b139b7de16d4c39acd5ba4d9ab8648d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f7acc1fba04ddebccc688885357742" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44367 } } peers { permanent_uuid: "8b139b7de16d4c39acd5ba4d9ab8648d" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36999 } } peers { permanent_uuid: "9664a73064234bc4912c8ddb99a59cb0" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43297 } }
I20260811 22:25:20.110028 629 raft_consensus.cc:374] T 2c2f3efbcd984f57b698448fdf2caa65 P 8b139b7de16d4c39acd5ba4d9ab8648d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:25:20.110152 629 raft_consensus.cc:729] T 2c2f3efbcd984f57b698448fdf2caa65 P 8b139b7de16d4c39acd5ba4d9ab8648d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b139b7de16d4c39acd5ba4d9ab8648d, State: Initialized, Role: FOLLOWER
I20260811 22:25:20.110754 629 consensus_queue.cc:260] T 2c2f3efbcd984f57b698448fdf2caa65 P 8b139b7de16d4c39acd5ba4d9ab8648d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f7acc1fba04ddebccc688885357742" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44367 } } peers { permanent_uuid: "8b139b7de16d4c39acd5ba4d9ab8648d" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36999 } } peers { permanent_uuid: "9664a73064234bc4912c8ddb99a59cb0" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43297 } }
I20260811 22:25:20.111881 627 raft_consensus.cc:348] T 2c2f3efbcd984f57b698448fdf2caa65 P 85f7acc1fba04ddebccc688885357742 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f7acc1fba04ddebccc688885357742" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44367 } } peers { permanent_uuid: "8b139b7de16d4c39acd5ba4d9ab8648d" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36999 } } peers { permanent_uuid: "9664a73064234bc4912c8ddb99a59cb0" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43297 } }
I20260811 22:25:20.112680 593 heartbeater.cc:503] Master 127.29.106.62:35421 was elected leader, sending a full tablet report...
I20260811 22:25:20.113482 629 ts_tablet_manager.cc:1428] T 2c2f3efbcd984f57b698448fdf2caa65 P 8b139b7de16d4c39acd5ba4d9ab8648d: Time spent starting tablet: real 0.015s user 0.012s sys 0.001s
I20260811 22:25:20.113716 625 ts_tablet_manager.cc:1428] T 2c2f3efbcd984f57b698448fdf2caa65 P 9664a73064234bc4912c8ddb99a59cb0: Time spent starting tablet: real 0.025s user 0.010s sys 0.014s
I20260811 22:25:20.112422 627 raft_consensus.cc:374] T 2c2f3efbcd984f57b698448fdf2caa65 P 85f7acc1fba04ddebccc688885357742 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:25:20.114190 627 raft_consensus.cc:729] T 2c2f3efbcd984f57b698448fdf2caa65 P 85f7acc1fba04ddebccc688885357742 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f7acc1fba04ddebccc688885357742, State: Initialized, Role: FOLLOWER
I20260811 22:25:20.114902 627 consensus_queue.cc:260] T 2c2f3efbcd984f57b698448fdf2caa65 P 85f7acc1fba04ddebccc688885357742 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f7acc1fba04ddebccc688885357742" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44367 } } peers { permanent_uuid: "8b139b7de16d4c39acd5ba4d9ab8648d" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36999 } } peers { permanent_uuid: "9664a73064234bc4912c8ddb99a59cb0" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43297 } }
I20260811 22:25:20.116580 349 heartbeater.cc:503] Master 127.29.106.62:35421 was elected leader, sending a full tablet report...
I20260811 22:25:20.116973 627 ts_tablet_manager.cc:1428] T 2c2f3efbcd984f57b698448fdf2caa65 P 85f7acc1fba04ddebccc688885357742: Time spent starting tablet: real 0.018s user 0.015s sys 0.000s
W20260811 22:25:20.183071 350 tablet.cc:2369] T 2c2f3efbcd984f57b698448fdf2caa65 P 85f7acc1fba04ddebccc688885357742: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 22:25:20.213152 32570 tablet.cc:2369] T 2c2f3efbcd984f57b698448fdf2caa65 P 9664a73064234bc4912c8ddb99a59cb0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 22:25:20.226707 594 tablet.cc:2369] T 2c2f3efbcd984f57b698448fdf2caa65 P 8b139b7de16d4c39acd5ba4d9ab8648d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:25:20.242528 640 raft_consensus.cc:482] T 2c2f3efbcd984f57b698448fdf2caa65 P 85f7acc1fba04ddebccc688885357742 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:25:20.243048 640 raft_consensus.cc:504] T 2c2f3efbcd984f57b698448fdf2caa65 P 85f7acc1fba04ddebccc688885357742 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f7acc1fba04ddebccc688885357742" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44367 } } peers { permanent_uuid: "8b139b7de16d4c39acd5ba4d9ab8648d" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36999 } } peers { permanent_uuid: "9664a73064234bc4912c8ddb99a59cb0" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43297 } }
I20260811 22:25:20.245565 640 leader_election.cc:288] T 2c2f3efbcd984f57b698448fdf2caa65 P 85f7acc1fba04ddebccc688885357742 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b139b7de16d4c39acd5ba4d9ab8648d (127.29.106.3:36999), 9664a73064234bc4912c8ddb99a59cb0 (127.29.106.1:43297)
I20260811 22:25:20.256541 620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:20.246114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:45981 (local address 127.29.106.1:43297)
0811 22:25:20.246328 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:20.246334 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:20.247692 (+ 1358us) server_negotiation.cc:407] Connection header received
0811 22:25:20.247959 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:20.247964 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:20.248056 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:20.248137 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:25:20.249392 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.250250 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.253942 (+ 3692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.255390 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.255550 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:20.255629 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:20.256119 (+ 490us) server_negotiation.cc:299] Negotiation successful
0811 22:25:20.256247 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260811 22:25:20.257058 644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:20.245784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:43297 (local address 127.29.106.2:45981)
0811 22:25:20.247574 (+ 1790us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:20.247603 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:20.247766 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:20.248183 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:20.248188 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:20.248241 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:25:20.249258 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:20.249274 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.250378 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.250382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:20.253780 (+ 3398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:20.253796 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.255746 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.255753 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:20.255959 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:20.255986 (+ 27us) client_negotiation.cc:769] Sending connection context
0811 22:25:20.256074 (+ 88us) client_negotiation.cc:240] Negotiation successful
0811 22:25:20.256218 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1638,"mutex_wait_us":24,"thread_start_us":1210,"threads_started":1}
I20260811 22:25:20.257596 623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:20.246194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:58907 (local address 127.29.106.3:36999)
0811 22:25:20.246491 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:20.246497 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:20.248788 (+ 2291us) server_negotiation.cc:407] Connection header received
0811 22:25:20.249005 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:20.249010 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:20.249091 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:20.249192 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:25:20.250350 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.251141 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.254772 (+ 3631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.256242 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.256524 (+ 282us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:20.256604 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:20.257054 (+ 450us) server_negotiation.cc:299] Negotiation successful
0811 22:25:20.257182 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
I20260811 22:25:20.258070 649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:20.246284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.3:36999 (local address 127.29.106.2:58907)
0811 22:25:20.248684 (+ 2400us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:20.248699 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:20.248821 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:20.249224 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:20.249228 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:20.249253 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:25:20.250168 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:20.250181 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.251348 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.251355 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:20.254624 (+ 3269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:20.254638 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.256628 (+ 1990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.256636 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:20.256853 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:20.256886 (+ 33us) client_negotiation.cc:769] Sending connection context
0811 22:25:20.257028 (+ 142us) client_negotiation.cc:240] Negotiation successful
0811 22:25:20.257312 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2142,"mutex_wait_us":1,"thread_start_us":1593,"threads_started":1}
I20260811 22:25:20.258273 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c2f3efbcd984f57b698448fdf2caa65" candidate_uuid: "85f7acc1fba04ddebccc688885357742" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9664a73064234bc4912c8ddb99a59cb0" is_pre_election: true
I20260811 22:25:20.259142 32466 raft_consensus.cc:2454] T 2c2f3efbcd984f57b698448fdf2caa65 P 9664a73064234bc4912c8ddb99a59cb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f7acc1fba04ddebccc688885357742 in term 0.
I20260811 22:25:20.259518 501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c2f3efbcd984f57b698448fdf2caa65" candidate_uuid: "85f7acc1fba04ddebccc688885357742" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b139b7de16d4c39acd5ba4d9ab8648d" is_pre_election: true
I20260811 22:25:20.260304 32603 leader_election.cc:302] T 2c2f3efbcd984f57b698448fdf2caa65 P 85f7acc1fba04ddebccc688885357742 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f7acc1fba04ddebccc688885357742, 9664a73064234bc4912c8ddb99a59cb0; no voters:
I20260811 22:25:20.260466 501 raft_consensus.cc:2454] T 2c2f3efbcd984f57b698448fdf2caa65 P 8b139b7de16d4c39acd5ba4d9ab8648d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85f7acc1fba04ddebccc688885357742 in term 0.
I20260811 22:25:20.260861 640 raft_consensus.cc:2781] T 2c2f3efbcd984f57b698448fdf2caa65 P 85f7acc1fba04ddebccc688885357742 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:25:20.261091 640 raft_consensus.cc:482] T 2c2f3efbcd984f57b698448fdf2caa65 P 85f7acc1fba04ddebccc688885357742 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:25:20.261278 640 raft_consensus.cc:3037] T 2c2f3efbcd984f57b698448fdf2caa65 P 85f7acc1fba04ddebccc688885357742 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:20.265154 640 raft_consensus.cc:504] T 2c2f3efbcd984f57b698448fdf2caa65 P 85f7acc1fba04ddebccc688885357742 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f7acc1fba04ddebccc688885357742" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44367 } } peers { permanent_uuid: "8b139b7de16d4c39acd5ba4d9ab8648d" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36999 } } peers { permanent_uuid: "9664a73064234bc4912c8ddb99a59cb0" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43297 } }
I20260811 22:25:20.266000 640 leader_election.cc:288] T 2c2f3efbcd984f57b698448fdf2caa65 P 85f7acc1fba04ddebccc688885357742 [CANDIDATE]: Term 1 election: Requested vote from peers 8b139b7de16d4c39acd5ba4d9ab8648d (127.29.106.3:36999), 9664a73064234bc4912c8ddb99a59cb0 (127.29.106.1:43297)
I20260811 22:25:20.266433 501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c2f3efbcd984f57b698448fdf2caa65" candidate_uuid: "85f7acc1fba04ddebccc688885357742" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b139b7de16d4c39acd5ba4d9ab8648d"
I20260811 22:25:20.266523 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c2f3efbcd984f57b698448fdf2caa65" candidate_uuid: "85f7acc1fba04ddebccc688885357742" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9664a73064234bc4912c8ddb99a59cb0"
I20260811 22:25:20.266688 501 raft_consensus.cc:3037] T 2c2f3efbcd984f57b698448fdf2caa65 P 8b139b7de16d4c39acd5ba4d9ab8648d [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:20.266777 32466 raft_consensus.cc:3037] T 2c2f3efbcd984f57b698448fdf2caa65 P 9664a73064234bc4912c8ddb99a59cb0 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:20.270041 501 raft_consensus.cc:2454] T 2c2f3efbcd984f57b698448fdf2caa65 P 8b139b7de16d4c39acd5ba4d9ab8648d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f7acc1fba04ddebccc688885357742 in term 1.
I20260811 22:25:20.270394 32466 raft_consensus.cc:2454] T 2c2f3efbcd984f57b698448fdf2caa65 P 9664a73064234bc4912c8ddb99a59cb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85f7acc1fba04ddebccc688885357742 in term 1.
I20260811 22:25:20.270612 32604 leader_election.cc:302] T 2c2f3efbcd984f57b698448fdf2caa65 P 85f7acc1fba04ddebccc688885357742 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f7acc1fba04ddebccc688885357742, 8b139b7de16d4c39acd5ba4d9ab8648d; no voters:
I20260811 22:25:20.271010 640 raft_consensus.cc:2781] T 2c2f3efbcd984f57b698448fdf2caa65 P 85f7acc1fba04ddebccc688885357742 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:25:20.271822 640 raft_consensus.cc:686] T 2c2f3efbcd984f57b698448fdf2caa65 P 85f7acc1fba04ddebccc688885357742 [term 1 LEADER]: Becoming Leader. State: Replica: 85f7acc1fba04ddebccc688885357742, State: Running, Role: LEADER
I20260811 22:25:20.272418 640 consensus_queue.cc:237] T 2c2f3efbcd984f57b698448fdf2caa65 P 85f7acc1fba04ddebccc688885357742 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f7acc1fba04ddebccc688885357742" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44367 } } peers { permanent_uuid: "8b139b7de16d4c39acd5ba4d9ab8648d" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36999 } } peers { permanent_uuid: "9664a73064234bc4912c8ddb99a59cb0" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43297 } }
I20260811 22:25:20.277920 32185 catalog_manager.cc:5495] T 2c2f3efbcd984f57b698448fdf2caa65 P 85f7acc1fba04ddebccc688885357742 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85f7acc1fba04ddebccc688885357742 (127.29.106.2). New cstate: current_term: 1 leader_uuid: "85f7acc1fba04ddebccc688885357742" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f7acc1fba04ddebccc688885357742" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b139b7de16d4c39acd5ba4d9ab8648d" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9664a73064234bc4912c8ddb99a59cb0" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43297 } health_report { overall_health: UNKNOWN } } }
I20260811 22:25:20.332665 618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:20.325213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40680 (local address 127.29.106.2:44367)
0811 22:25:20.325423 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:20.325428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:20.325720 (+ 292us) server_negotiation.cc:407] Connection header received
0811 22:25:20.326149 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:20.326153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:20.326235 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:20.326362 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:25:20.326884 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.327933 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.329727 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.330408 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.330559 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:20.331345 (+ 786us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:25:20.332001 (+ 656us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:25:20.332109 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:20.332226 (+ 117us) server_negotiation.cc:299] Negotiation successful
0811 22:25:20.332310 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260811 22:25:20.344799 32466 raft_consensus.cc:1261] T 2c2f3efbcd984f57b698448fdf2caa65 P 9664a73064234bc4912c8ddb99a59cb0 [term 1 FOLLOWER]: Refusing update from remote peer 85f7acc1fba04ddebccc688885357742: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:25:20.346335 652 consensus_queue.cc:1035] T 2c2f3efbcd984f57b698448fdf2caa65 P 85f7acc1fba04ddebccc688885357742 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9664a73064234bc4912c8ddb99a59cb0" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:25:20.347014 501 raft_consensus.cc:1261] T 2c2f3efbcd984f57b698448fdf2caa65 P 8b139b7de16d4c39acd5ba4d9ab8648d [term 1 FOLLOWER]: Refusing update from remote peer 85f7acc1fba04ddebccc688885357742: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:25:20.348069 652 consensus_queue.cc:1035] T 2c2f3efbcd984f57b698448fdf2caa65 P 85f7acc1fba04ddebccc688885357742 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b139b7de16d4c39acd5ba4d9ab8648d" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:25:20.382275 659 mvcc.cc:205] Tried to move back new op lower bound from 7317451244918423552 to 7317451244640292864. Current Snapshot: MvccSnapshot[applied={T|T < 7317451244918423552}]
I20260811 22:25:20.398481 32185 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:25:20.398924 32185 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:25:20.404018 32185 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c2fed88d6bdc4913bd6dd4e7d89e1279: Sending DeleteTablet for 3 replicas of tablet 2c2f3efbcd984f57b698448fdf2caa65
I20260811 22:25:20.405663 463 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c2f3efbcd984f57b698448fdf2caa65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:25:20 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:39716
I20260811 22:25:20.405616 32686 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c2f3efbcd984f57b698448fdf2caa65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:25:20 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:40674
I20260811 22:25:20.405655 32428 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c2f3efbcd984f57b698448fdf2caa65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:25:20 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:51108
I20260811 22:25:20.408746 32183 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41556:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:25:20.409723 32183 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:25:20.419064 32426 tablet_service.cc:1467] Processing CreateTablet for tablet a6125ea6a8754dc19cf98ff177382e05 (DEFAULT_TABLE table=test-table [id=2fcb37bd8b0542339d95edff45b9add5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:25:20.420106 32426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6125ea6a8754dc19cf98ff177382e05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:20.420014 32684 tablet_service.cc:1467] Processing CreateTablet for tablet a6125ea6a8754dc19cf98ff177382e05 (DEFAULT_TABLE table=test-table [id=2fcb37bd8b0542339d95edff45b9add5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:25:20.421160 32684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6125ea6a8754dc19cf98ff177382e05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:20.424216 462 tablet_service.cc:1467] Processing CreateTablet for tablet a6125ea6a8754dc19cf98ff177382e05 (DEFAULT_TABLE table=test-table [id=2fcb37bd8b0542339d95edff45b9add5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:25:20.425424 462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6125ea6a8754dc19cf98ff177382e05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:20.431248 678 tablet_replica.cc:330] T 2c2f3efbcd984f57b698448fdf2caa65 P 8b139b7de16d4c39acd5ba4d9ab8648d: stopping tablet replica
I20260811 22:25:20.433948 683 tablet_replica.cc:330] T 2c2f3efbcd984f57b698448fdf2caa65 P 85f7acc1fba04ddebccc688885357742: stopping tablet replica
I20260811 22:25:20.434643 683 raft_consensus.cc:2229] T 2c2f3efbcd984f57b698448fdf2caa65 P 85f7acc1fba04ddebccc688885357742 [term 1 LEADER]: Raft consensus shutting down.
I20260811 22:25:20.436498 678 raft_consensus.cc:2229] T 2c2f3efbcd984f57b698448fdf2caa65 P 8b139b7de16d4c39acd5ba4d9ab8648d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:25:20.437425 678 raft_consensus.cc:2258] T 2c2f3efbcd984f57b698448fdf2caa65 P 8b139b7de16d4c39acd5ba4d9ab8648d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:25:20.439054 679 tablet_replica.cc:330] T 2c2f3efbcd984f57b698448fdf2caa65 P 9664a73064234bc4912c8ddb99a59cb0: stopping tablet replica
I20260811 22:25:20.439378 627 tablet_bootstrap.cc:492] T a6125ea6a8754dc19cf98ff177382e05 P 85f7acc1fba04ddebccc688885357742: Bootstrap starting.
I20260811 22:25:20.440912 625 tablet_bootstrap.cc:492] T a6125ea6a8754dc19cf98ff177382e05 P 9664a73064234bc4912c8ddb99a59cb0: Bootstrap starting.
I20260811 22:25:20.442494 629 tablet_bootstrap.cc:492] T a6125ea6a8754dc19cf98ff177382e05 P 8b139b7de16d4c39acd5ba4d9ab8648d: Bootstrap starting.
I20260811 22:25:20.443658 625 tablet_bootstrap.cc:654] T a6125ea6a8754dc19cf98ff177382e05 P 9664a73064234bc4912c8ddb99a59cb0: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:20.443848 627 tablet_bootstrap.cc:654] T a6125ea6a8754dc19cf98ff177382e05 P 85f7acc1fba04ddebccc688885357742: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:20.445145 629 tablet_bootstrap.cc:654] T a6125ea6a8754dc19cf98ff177382e05 P 8b139b7de16d4c39acd5ba4d9ab8648d: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:20.446929 678 ts_tablet_manager.cc:1905] T 2c2f3efbcd984f57b698448fdf2caa65 P 8b139b7de16d4c39acd5ba4d9ab8648d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:25:20.448098 625 tablet_bootstrap.cc:492] T a6125ea6a8754dc19cf98ff177382e05 P 9664a73064234bc4912c8ddb99a59cb0: No bootstrap required, opened a new log
I20260811 22:25:20.448308 625 ts_tablet_manager.cc:1397] T a6125ea6a8754dc19cf98ff177382e05 P 9664a73064234bc4912c8ddb99a59cb0: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260811 22:25:20.449225 625 raft_consensus.cc:348] T a6125ea6a8754dc19cf98ff177382e05 P 9664a73064234bc4912c8ddb99a59cb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9664a73064234bc4912c8ddb99a59cb0" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43297 } } peers { permanent_uuid: "85f7acc1fba04ddebccc688885357742" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44367 } } peers { permanent_uuid: "8b139b7de16d4c39acd5ba4d9ab8648d" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36999 } }
I20260811 22:25:20.449486 625 raft_consensus.cc:374] T a6125ea6a8754dc19cf98ff177382e05 P 9664a73064234bc4912c8ddb99a59cb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:25:20.449580 625 raft_consensus.cc:729] T a6125ea6a8754dc19cf98ff177382e05 P 9664a73064234bc4912c8ddb99a59cb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9664a73064234bc4912c8ddb99a59cb0, State: Initialized, Role: FOLLOWER
I20260811 22:25:20.449853 625 consensus_queue.cc:260] T a6125ea6a8754dc19cf98ff177382e05 P 9664a73064234bc4912c8ddb99a59cb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9664a73064234bc4912c8ddb99a59cb0" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43297 } } peers { permanent_uuid: "85f7acc1fba04ddebccc688885357742" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44367 } } peers { permanent_uuid: "8b139b7de16d4c39acd5ba4d9ab8648d" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36999 } }
I20260811 22:25:20.450428 625 ts_tablet_manager.cc:1428] T a6125ea6a8754dc19cf98ff177382e05 P 9664a73064234bc4912c8ddb99a59cb0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260811 22:25:20.450816 683 raft_consensus.cc:2258] T 2c2f3efbcd984f57b698448fdf2caa65 P 85f7acc1fba04ddebccc688885357742 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:25:20.452661 683 ts_tablet_manager.cc:1905] T 2c2f3efbcd984f57b698448fdf2caa65 P 85f7acc1fba04ddebccc688885357742: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:25:20.440920 679 raft_consensus.cc:2229] T 2c2f3efbcd984f57b698448fdf2caa65 P 9664a73064234bc4912c8ddb99a59cb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:25:20.456338 679 raft_consensus.cc:2258] T 2c2f3efbcd984f57b698448fdf2caa65 P 9664a73064234bc4912c8ddb99a59cb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:25:20.458820 683 ts_tablet_manager.cc:1918] T 2c2f3efbcd984f57b698448fdf2caa65 P 85f7acc1fba04ddebccc688885357742: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:25:20.459004 683 log.cc:1196] T 2c2f3efbcd984f57b698448fdf2caa65 P 85f7acc1fba04ddebccc688885357742: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-1/wal/wals/2c2f3efbcd984f57b698448fdf2caa65
I20260811 22:25:20.460897 679 ts_tablet_manager.cc:1905] T 2c2f3efbcd984f57b698448fdf2caa65 P 9664a73064234bc4912c8ddb99a59cb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:25:20.463825 683 ts_tablet_manager.cc:1939] T 2c2f3efbcd984f57b698448fdf2caa65 P 85f7acc1fba04ddebccc688885357742: Deleting consensus metadata
I20260811 22:25:20.465764 32159 catalog_manager.cc:4841] TS 85f7acc1fba04ddebccc688885357742 (127.29.106.2:44367): tablet 2c2f3efbcd984f57b698448fdf2caa65 (table test-table [id=45f361ad02ba4525ab8449020f08aad0]) successfully deleted
I20260811 22:25:20.467689 627 tablet_bootstrap.cc:492] T a6125ea6a8754dc19cf98ff177382e05 P 85f7acc1fba04ddebccc688885357742: No bootstrap required, opened a new log
I20260811 22:25:20.467836 627 ts_tablet_manager.cc:1397] T a6125ea6a8754dc19cf98ff177382e05 P 85f7acc1fba04ddebccc688885357742: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.004s
I20260811 22:25:20.468025 678 ts_tablet_manager.cc:1918] T 2c2f3efbcd984f57b698448fdf2caa65 P 8b139b7de16d4c39acd5ba4d9ab8648d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:25:20.468214 678 log.cc:1196] T 2c2f3efbcd984f57b698448fdf2caa65 P 8b139b7de16d4c39acd5ba4d9ab8648d: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-2/wal/wals/2c2f3efbcd984f57b698448fdf2caa65
I20260811 22:25:20.468740 678 ts_tablet_manager.cc:1939] T 2c2f3efbcd984f57b698448fdf2caa65 P 8b139b7de16d4c39acd5ba4d9ab8648d: Deleting consensus metadata
I20260811 22:25:20.468616 627 raft_consensus.cc:348] T a6125ea6a8754dc19cf98ff177382e05 P 85f7acc1fba04ddebccc688885357742 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9664a73064234bc4912c8ddb99a59cb0" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43297 } } peers { permanent_uuid: "85f7acc1fba04ddebccc688885357742" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44367 } } peers { permanent_uuid: "8b139b7de16d4c39acd5ba4d9ab8648d" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36999 } }
I20260811 22:25:20.468844 627 raft_consensus.cc:374] T a6125ea6a8754dc19cf98ff177382e05 P 85f7acc1fba04ddebccc688885357742 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:25:20.468950 627 raft_consensus.cc:729] T a6125ea6a8754dc19cf98ff177382e05 P 85f7acc1fba04ddebccc688885357742 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f7acc1fba04ddebccc688885357742, State: Initialized, Role: FOLLOWER
I20260811 22:25:20.469210 627 consensus_queue.cc:260] T a6125ea6a8754dc19cf98ff177382e05 P 85f7acc1fba04ddebccc688885357742 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9664a73064234bc4912c8ddb99a59cb0" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43297 } } peers { permanent_uuid: "85f7acc1fba04ddebccc688885357742" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44367 } } peers { permanent_uuid: "8b139b7de16d4c39acd5ba4d9ab8648d" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36999 } }
I20260811 22:25:20.469775 627 ts_tablet_manager.cc:1428] T a6125ea6a8754dc19cf98ff177382e05 P 85f7acc1fba04ddebccc688885357742: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260811 22:25:20.471369 32159 catalog_manager.cc:4841] TS 8b139b7de16d4c39acd5ba4d9ab8648d (127.29.106.3:36999): tablet 2c2f3efbcd984f57b698448fdf2caa65 (table test-table [id=45f361ad02ba4525ab8449020f08aad0]) successfully deleted
I20260811 22:25:20.472254 679 ts_tablet_manager.cc:1918] T 2c2f3efbcd984f57b698448fdf2caa65 P 9664a73064234bc4912c8ddb99a59cb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:25:20.472409 679 log.cc:1196] T 2c2f3efbcd984f57b698448fdf2caa65 P 9664a73064234bc4912c8ddb99a59cb0: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-0/wal/wals/2c2f3efbcd984f57b698448fdf2caa65
I20260811 22:25:20.472913 679 ts_tablet_manager.cc:1939] T 2c2f3efbcd984f57b698448fdf2caa65 P 9664a73064234bc4912c8ddb99a59cb0: Deleting consensus metadata
I20260811 22:25:20.474491 32158 catalog_manager.cc:4841] TS 9664a73064234bc4912c8ddb99a59cb0 (127.29.106.1:43297): tablet 2c2f3efbcd984f57b698448fdf2caa65 (table test-table [id=45f361ad02ba4525ab8449020f08aad0]) successfully deleted
I20260811 22:25:20.475374 629 tablet_bootstrap.cc:492] T a6125ea6a8754dc19cf98ff177382e05 P 8b139b7de16d4c39acd5ba4d9ab8648d: No bootstrap required, opened a new log
I20260811 22:25:20.475524 629 ts_tablet_manager.cc:1397] T a6125ea6a8754dc19cf98ff177382e05 P 8b139b7de16d4c39acd5ba4d9ab8648d: Time spent bootstrapping tablet: real 0.033s user 0.010s sys 0.019s
I20260811 22:25:20.476246 629 raft_consensus.cc:348] T a6125ea6a8754dc19cf98ff177382e05 P 8b139b7de16d4c39acd5ba4d9ab8648d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9664a73064234bc4912c8ddb99a59cb0" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43297 } } peers { permanent_uuid: "85f7acc1fba04ddebccc688885357742" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44367 } } peers { permanent_uuid: "8b139b7de16d4c39acd5ba4d9ab8648d" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36999 } }
I20260811 22:25:20.476480 629 raft_consensus.cc:374] T a6125ea6a8754dc19cf98ff177382e05 P 8b139b7de16d4c39acd5ba4d9ab8648d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:25:20.476574 629 raft_consensus.cc:729] T a6125ea6a8754dc19cf98ff177382e05 P 8b139b7de16d4c39acd5ba4d9ab8648d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b139b7de16d4c39acd5ba4d9ab8648d, State: Initialized, Role: FOLLOWER
I20260811 22:25:20.476825 629 consensus_queue.cc:260] T a6125ea6a8754dc19cf98ff177382e05 P 8b139b7de16d4c39acd5ba4d9ab8648d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9664a73064234bc4912c8ddb99a59cb0" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43297 } } peers { permanent_uuid: "85f7acc1fba04ddebccc688885357742" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44367 } } peers { permanent_uuid: "8b139b7de16d4c39acd5ba4d9ab8648d" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36999 } }
I20260811 22:25:20.477387 629 ts_tablet_manager.cc:1428] T a6125ea6a8754dc19cf98ff177382e05 P 8b139b7de16d4c39acd5ba4d9ab8648d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260811 22:25:20.563385 638 raft_consensus.cc:482] T a6125ea6a8754dc19cf98ff177382e05 P 8b139b7de16d4c39acd5ba4d9ab8648d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:25:20.563621 638 raft_consensus.cc:504] T a6125ea6a8754dc19cf98ff177382e05 P 8b139b7de16d4c39acd5ba4d9ab8648d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9664a73064234bc4912c8ddb99a59cb0" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43297 } } peers { permanent_uuid: "85f7acc1fba04ddebccc688885357742" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44367 } } peers { permanent_uuid: "8b139b7de16d4c39acd5ba4d9ab8648d" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36999 } }
I20260811 22:25:20.565122 638 leader_election.cc:288] T a6125ea6a8754dc19cf98ff177382e05 P 8b139b7de16d4c39acd5ba4d9ab8648d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9664a73064234bc4912c8ddb99a59cb0 (127.29.106.1:43297), 85f7acc1fba04ddebccc688885357742 (127.29.106.2:44367)
I20260811 22:25:20.574901 620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:20.565708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:52979 (local address 127.29.106.1:43297)
0811 22:25:20.565879 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:20.565883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:20.566138 (+ 255us) server_negotiation.cc:407] Connection header received
0811 22:25:20.566427 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:20.566433 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:20.566504 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:20.566572 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:25:20.567404 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.568794 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.571884 (+ 3090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.573477 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.573624 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:20.573690 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:20.574228 (+ 538us) server_negotiation.cc:299] Negotiation successful
0811 22:25:20.574356 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260811 22:25:20.576315 692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:20.565306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:43297 (local address 127.29.106.3:52979)
0811 22:25:20.566027 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:20.566046 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:20.566252 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:20.566812 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:20.566817 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:20.566863 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:25:20.567130 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:20.567143 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.569017 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.569022 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:20.571679 (+ 2657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:20.571696 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.573832 (+ 2136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.573837 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:20.574042 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:20.574078 (+ 36us) client_negotiation.cc:769] Sending connection context
0811 22:25:20.575436 (+ 1358us) client_negotiation.cc:240] Negotiation successful
0811 22:25:20.575668 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":576,"threads_started":1}
I20260811 22:25:20.579190 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6125ea6a8754dc19cf98ff177382e05" candidate_uuid: "8b139b7de16d4c39acd5ba4d9ab8648d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9664a73064234bc4912c8ddb99a59cb0" is_pre_election: true
I20260811 22:25:20.579504 32466 raft_consensus.cc:2454] T a6125ea6a8754dc19cf98ff177382e05 P 9664a73064234bc4912c8ddb99a59cb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b139b7de16d4c39acd5ba4d9ab8648d in term 0.
I20260811 22:25:20.580179 385 leader_election.cc:302] T a6125ea6a8754dc19cf98ff177382e05 P 8b139b7de16d4c39acd5ba4d9ab8648d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b139b7de16d4c39acd5ba4d9ab8648d, 9664a73064234bc4912c8ddb99a59cb0; no voters:
I20260811 22:25:20.580657 638 raft_consensus.cc:2781] T a6125ea6a8754dc19cf98ff177382e05 P 8b139b7de16d4c39acd5ba4d9ab8648d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:25:20.580780 638 raft_consensus.cc:482] T a6125ea6a8754dc19cf98ff177382e05 P 8b139b7de16d4c39acd5ba4d9ab8648d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:25:20.580904 638 raft_consensus.cc:3037] T a6125ea6a8754dc19cf98ff177382e05 P 8b139b7de16d4c39acd5ba4d9ab8648d [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:20.584857 692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:20.565462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.2:44367 (local address 127.29.106.3:36851)
0811 22:25:20.576666 (+ 11204us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:20.576731 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:20.576894 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:20.577459 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:20.577463 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:20.577492 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:25:20.577756 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:20.577770 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.580013 (+ 2243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.580017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:20.582051 (+ 2034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:20.582064 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.583934 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.583941 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:20.584243 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:20.584272 (+ 29us) client_negotiation.cc:769] Sending connection context
0811 22:25:20.584350 (+ 78us) client_negotiation.cc:240] Negotiation successful
0811 22:25:20.584518 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10991,"thread_start_us":15367,"threads_started":1}
I20260811 22:25:20.585024 618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:20.565487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:36851 (local address 127.29.106.2:44367)
0811 22:25:20.565675 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:20.565682 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:20.576797 (+ 11115us) server_negotiation.cc:407] Connection header received
0811 22:25:20.577056 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:20.577063 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:20.577149 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:20.577222 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:25:20.577936 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.579700 (+ 1764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.582272 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.583674 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.583814 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:20.583970 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:20.584409 (+ 439us) server_negotiation.cc:299] Negotiation successful
0811 22:25:20.584542 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260811 22:25:20.585681 638 raft_consensus.cc:504] T a6125ea6a8754dc19cf98ff177382e05 P 8b139b7de16d4c39acd5ba4d9ab8648d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9664a73064234bc4912c8ddb99a59cb0" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43297 } } peers { permanent_uuid: "85f7acc1fba04ddebccc688885357742" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44367 } } peers { permanent_uuid: "8b139b7de16d4c39acd5ba4d9ab8648d" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36999 } }
I20260811 22:25:20.586261 32722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6125ea6a8754dc19cf98ff177382e05" candidate_uuid: "8b139b7de16d4c39acd5ba4d9ab8648d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f7acc1fba04ddebccc688885357742" is_pre_election: true
I20260811 22:25:20.586763 638 leader_election.cc:288] T a6125ea6a8754dc19cf98ff177382e05 P 8b139b7de16d4c39acd5ba4d9ab8648d [CANDIDATE]: Term 1 election: Requested vote from peers 9664a73064234bc4912c8ddb99a59cb0 (127.29.106.1:43297), 85f7acc1fba04ddebccc688885357742 (127.29.106.2:44367)
I20260811 22:25:20.587054 32722 raft_consensus.cc:2454] T a6125ea6a8754dc19cf98ff177382e05 P 85f7acc1fba04ddebccc688885357742 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b139b7de16d4c39acd5ba4d9ab8648d in term 0.
I20260811 22:25:20.587340 32466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6125ea6a8754dc19cf98ff177382e05" candidate_uuid: "8b139b7de16d4c39acd5ba4d9ab8648d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9664a73064234bc4912c8ddb99a59cb0"
I20260811 22:25:20.587653 32466 raft_consensus.cc:3037] T a6125ea6a8754dc19cf98ff177382e05 P 9664a73064234bc4912c8ddb99a59cb0 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:20.588088 32721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6125ea6a8754dc19cf98ff177382e05" candidate_uuid: "8b139b7de16d4c39acd5ba4d9ab8648d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f7acc1fba04ddebccc688885357742"
I20260811 22:25:20.588532 32721 raft_consensus.cc:3037] T a6125ea6a8754dc19cf98ff177382e05 P 85f7acc1fba04ddebccc688885357742 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:20.591954 32466 raft_consensus.cc:2454] T a6125ea6a8754dc19cf98ff177382e05 P 9664a73064234bc4912c8ddb99a59cb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b139b7de16d4c39acd5ba4d9ab8648d in term 1.
I20260811 22:25:20.592428 385 leader_election.cc:302] T a6125ea6a8754dc19cf98ff177382e05 P 8b139b7de16d4c39acd5ba4d9ab8648d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b139b7de16d4c39acd5ba4d9ab8648d, 9664a73064234bc4912c8ddb99a59cb0; no voters:
I20260811 22:25:20.592941 638 raft_consensus.cc:2781] T a6125ea6a8754dc19cf98ff177382e05 P 8b139b7de16d4c39acd5ba4d9ab8648d [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:25:20.593954 638 raft_consensus.cc:686] T a6125ea6a8754dc19cf98ff177382e05 P 8b139b7de16d4c39acd5ba4d9ab8648d [term 1 LEADER]: Becoming Leader. State: Replica: 8b139b7de16d4c39acd5ba4d9ab8648d, State: Running, Role: LEADER
I20260811 22:25:20.594252 32721 raft_consensus.cc:2454] T a6125ea6a8754dc19cf98ff177382e05 P 85f7acc1fba04ddebccc688885357742 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b139b7de16d4c39acd5ba4d9ab8648d in term 1.
I20260811 22:25:20.594558 638 consensus_queue.cc:237] T a6125ea6a8754dc19cf98ff177382e05 P 8b139b7de16d4c39acd5ba4d9ab8648d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9664a73064234bc4912c8ddb99a59cb0" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43297 } } peers { permanent_uuid: "85f7acc1fba04ddebccc688885357742" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44367 } } peers { permanent_uuid: "8b139b7de16d4c39acd5ba4d9ab8648d" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36999 } }
I20260811 22:25:20.600068 32183 catalog_manager.cc:5495] T a6125ea6a8754dc19cf98ff177382e05 P 8b139b7de16d4c39acd5ba4d9ab8648d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b139b7de16d4c39acd5ba4d9ab8648d (127.29.106.3). New cstate: current_term: 1 leader_uuid: "8b139b7de16d4c39acd5ba4d9ab8648d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9664a73064234bc4912c8ddb99a59cb0" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85f7acc1fba04ddebccc688885357742" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b139b7de16d4c39acd5ba4d9ab8648d" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36999 } health_report { overall_health: HEALTHY } } }
Aug 11 22:25:20 dist-test-slave-gjrz krb5kdc[32108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487118, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:25:20.671103 713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:20.646212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41566 (local address 127.29.106.62:35421)
0811 22:25:20.647513 (+ 1301us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:20.647520 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:20.648218 (+ 698us) server_negotiation.cc:407] Connection header received
0811 22:25:20.648380 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:20.648385 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:20.648471 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:20.648595 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:20.650513 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.652384 (+ 1871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.654275 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.655464 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.655596 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:20.663245 (+ 7649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:20.663252 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:20.663255 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:20.663336 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:20.667394 (+ 4058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:20.668462 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:20.668467 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:20.668470 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:20.668643 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:20.669264 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:20.669268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:20.669270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:20.669601 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:20.669702 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:20.670343 (+ 641us) server_negotiation.cc:299] Negotiation successful
0811 22:25:20.670661 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":128,"server-negotiator.queue_time_us":1059,"thread_start_us":738,"threads_started":1}
I20260811 22:25:20.671195 714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:20.646374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:35421 (local address 127.0.0.1:41566)
0811 22:25:20.648118 (+ 1744us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:20.648134 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:20.648276 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:20.648605 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:20.648608 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:20.649899 (+ 1291us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:20.650308 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:20.650320 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.652560 (+ 2240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.652564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:20.654128 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:20.654141 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.655645 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.655651 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:20.655872 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:20.659435 (+ 3563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:20.659452 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:20.663008 (+ 3556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:20.667514 (+ 4506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:20.667526 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:20.667534 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:20.668353 (+ 819us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:20.668764 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:20.668769 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:20.668773 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:20.669117 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:20.669753 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:20.669762 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:20.670022 (+ 260us) client_negotiation.cc:769] Sending connection context
0811 22:25:20.670215 (+ 193us) client_negotiation.cc:240] Negotiation successful
0811 22:25:20.670532 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1584,"thread_start_us":1349,"threads_started":1}
I20260811 22:25:20.677913 32183 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:41566:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260811 22:25:20.679344 32183 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:25:20.687326 32426 tablet_service.cc:1467] Processing CreateTablet for tablet 1e16074c04f643e5b4a36d2ebf1cb3eb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0ea5ee8e134a4cffb06ca692ea3ec4bf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 22:25:20.687390 32684 tablet_service.cc:1467] Processing CreateTablet for tablet 1e16074c04f643e5b4a36d2ebf1cb3eb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0ea5ee8e134a4cffb06ca692ea3ec4bf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 22:25:20.687958 32426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e16074c04f643e5b4a36d2ebf1cb3eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:20.688103 32684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e16074c04f643e5b4a36d2ebf1cb3eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:20.688720 462 tablet_service.cc:1467] Processing CreateTablet for tablet 1e16074c04f643e5b4a36d2ebf1cb3eb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0ea5ee8e134a4cffb06ca692ea3ec4bf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 22:25:20.689385 462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e16074c04f643e5b4a36d2ebf1cb3eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:20.698133 627 tablet_bootstrap.cc:492] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 85f7acc1fba04ddebccc688885357742: Bootstrap starting.
I20260811 22:25:20.698314 625 tablet_bootstrap.cc:492] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 9664a73064234bc4912c8ddb99a59cb0: Bootstrap starting.
I20260811 22:25:20.699214 629 tablet_bootstrap.cc:492] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 8b139b7de16d4c39acd5ba4d9ab8648d: Bootstrap starting.
I20260811 22:25:20.700908 627 tablet_bootstrap.cc:654] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 85f7acc1fba04ddebccc688885357742: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:20.701279 625 tablet_bootstrap.cc:654] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 9664a73064234bc4912c8ddb99a59cb0: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:20.702463 629 tablet_bootstrap.cc:654] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 8b139b7de16d4c39acd5ba4d9ab8648d: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:20.712209 627 tablet_bootstrap.cc:492] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 85f7acc1fba04ddebccc688885357742: No bootstrap required, opened a new log
I20260811 22:25:20.712468 625 tablet_bootstrap.cc:492] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 9664a73064234bc4912c8ddb99a59cb0: No bootstrap required, opened a new log
I20260811 22:25:20.712591 627 ts_tablet_manager.cc:1397] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 85f7acc1fba04ddebccc688885357742: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.008s
I20260811 22:25:20.712872 625 ts_tablet_manager.cc:1397] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 9664a73064234bc4912c8ddb99a59cb0: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.012s
I20260811 22:25:20.713454 629 tablet_bootstrap.cc:492] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 8b139b7de16d4c39acd5ba4d9ab8648d: No bootstrap required, opened a new log
I20260811 22:25:20.713801 629 ts_tablet_manager.cc:1397] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 8b139b7de16d4c39acd5ba4d9ab8648d: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.007s
I20260811 22:25:20.713909 627 raft_consensus.cc:348] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 85f7acc1fba04ddebccc688885357742 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9664a73064234bc4912c8ddb99a59cb0" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43297 } } peers { permanent_uuid: "85f7acc1fba04ddebccc688885357742" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44367 } } peers { permanent_uuid: "8b139b7de16d4c39acd5ba4d9ab8648d" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36999 } }
I20260811 22:25:20.714309 627 raft_consensus.cc:374] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 85f7acc1fba04ddebccc688885357742 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:25:20.714509 627 raft_consensus.cc:729] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 85f7acc1fba04ddebccc688885357742 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f7acc1fba04ddebccc688885357742, State: Initialized, Role: FOLLOWER
I20260811 22:25:20.714892 625 raft_consensus.cc:348] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 9664a73064234bc4912c8ddb99a59cb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9664a73064234bc4912c8ddb99a59cb0" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43297 } } peers { permanent_uuid: "85f7acc1fba04ddebccc688885357742" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44367 } } peers { permanent_uuid: "8b139b7de16d4c39acd5ba4d9ab8648d" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36999 } }
I20260811 22:25:20.714967 629 raft_consensus.cc:348] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 8b139b7de16d4c39acd5ba4d9ab8648d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9664a73064234bc4912c8ddb99a59cb0" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43297 } } peers { permanent_uuid: "85f7acc1fba04ddebccc688885357742" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44367 } } peers { permanent_uuid: "8b139b7de16d4c39acd5ba4d9ab8648d" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36999 } }
I20260811 22:25:20.715013 627 consensus_queue.cc:260] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 85f7acc1fba04ddebccc688885357742 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9664a73064234bc4912c8ddb99a59cb0" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43297 } } peers { permanent_uuid: "85f7acc1fba04ddebccc688885357742" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44367 } } peers { permanent_uuid: "8b139b7de16d4c39acd5ba4d9ab8648d" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36999 } }
I20260811 22:25:20.715349 625 raft_consensus.cc:374] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 9664a73064234bc4912c8ddb99a59cb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:25:20.715541 629 raft_consensus.cc:374] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 8b139b7de16d4c39acd5ba4d9ab8648d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:25:20.715562 625 raft_consensus.cc:729] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 9664a73064234bc4912c8ddb99a59cb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9664a73064234bc4912c8ddb99a59cb0, State: Initialized, Role: FOLLOWER
I20260811 22:25:20.715796 629 raft_consensus.cc:729] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 8b139b7de16d4c39acd5ba4d9ab8648d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b139b7de16d4c39acd5ba4d9ab8648d, State: Initialized, Role: FOLLOWER
I20260811 22:25:20.716126 625 consensus_queue.cc:260] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 9664a73064234bc4912c8ddb99a59cb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9664a73064234bc4912c8ddb99a59cb0" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43297 } } peers { permanent_uuid: "85f7acc1fba04ddebccc688885357742" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44367 } } peers { permanent_uuid: "8b139b7de16d4c39acd5ba4d9ab8648d" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36999 } }
I20260811 22:25:20.716236 629 consensus_queue.cc:260] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 8b139b7de16d4c39acd5ba4d9ab8648d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9664a73064234bc4912c8ddb99a59cb0" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43297 } } peers { permanent_uuid: "85f7acc1fba04ddebccc688885357742" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44367 } } peers { permanent_uuid: "8b139b7de16d4c39acd5ba4d9ab8648d" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36999 } }
I20260811 22:25:20.716871 638 tablet_replica.cc:439] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 8b139b7de16d4c39acd5ba4d9ab8648d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9664a73064234bc4912c8ddb99a59cb0" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43297 } } peers { permanent_uuid: "85f7acc1fba04ddebccc688885357742" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44367 } } peers { permanent_uuid: "8b139b7de16d4c39acd5ba4d9ab8648d" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36999 } } }
I20260811 22:25:20.717487 638 tablet_replica.cc:442] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 8b139b7de16d4c39acd5ba4d9ab8648d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 22:25:20.716815 637 tablet_replica.cc:439] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 9664a73064234bc4912c8ddb99a59cb0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9664a73064234bc4912c8ddb99a59cb0" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43297 } } peers { permanent_uuid: "85f7acc1fba04ddebccc688885357742" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44367 } } peers { permanent_uuid: "8b139b7de16d4c39acd5ba4d9ab8648d" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36999 } } }
I20260811 22:25:20.717687 637 tablet_replica.cc:442] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 9664a73064234bc4912c8ddb99a59cb0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 22:25:20.718206 625 ts_tablet_manager.cc:1428] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 9664a73064234bc4912c8ddb99a59cb0: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260811 22:25:20.715808 667 tablet_replica.cc:439] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 85f7acc1fba04ddebccc688885357742: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9664a73064234bc4912c8ddb99a59cb0" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43297 } } peers { permanent_uuid: "85f7acc1fba04ddebccc688885357742" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44367 } } peers { permanent_uuid: "8b139b7de16d4c39acd5ba4d9ab8648d" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36999 } } }
I20260811 22:25:20.718729 667 tablet_replica.cc:442] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 85f7acc1fba04ddebccc688885357742: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 22:25:20.720175 627 ts_tablet_manager.cc:1428] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 85f7acc1fba04ddebccc688885357742: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20260811 22:25:20.721297 629 ts_tablet_manager.cc:1428] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 8b139b7de16d4c39acd5ba4d9ab8648d: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20260811 22:25:20.726758 637 raft_consensus.cc:482] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 9664a73064234bc4912c8ddb99a59cb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:25:20.727183 637 raft_consensus.cc:504] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 9664a73064234bc4912c8ddb99a59cb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9664a73064234bc4912c8ddb99a59cb0" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43297 } } peers { permanent_uuid: "85f7acc1fba04ddebccc688885357742" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44367 } } peers { permanent_uuid: "8b139b7de16d4c39acd5ba4d9ab8648d" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36999 } }
I20260811 22:25:20.729318 637 leader_election.cc:288] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 9664a73064234bc4912c8ddb99a59cb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f7acc1fba04ddebccc688885357742 (127.29.106.2:44367), 8b139b7de16d4c39acd5ba4d9ab8648d (127.29.106.3:36999)
I20260811 22:25:20.741638 618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:20.729949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:34209 (local address 127.29.106.2:44367)
0811 22:25:20.730173 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:20.730178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:20.731088 (+ 910us) server_negotiation.cc:407] Connection header received
0811 22:25:20.731377 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:20.731383 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:20.731455 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:20.731533 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:25:20.733313 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.734930 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.738683 (+ 3753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.740216 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.740463 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:20.740689 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:20.741088 (+ 399us) server_negotiation.cc:299] Negotiation successful
0811 22:25:20.741243 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260811 22:25:20.741902 721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:20.729919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.2:44367 (local address 127.29.106.1:34209)
0811 22:25:20.730993 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:20.731009 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:20.731193 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:20.731875 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:20.731881 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:20.731911 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:25:20.733113 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:20.733126 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.735161 (+ 2035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.735165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:20.738426 (+ 3261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:20.738440 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.740619 (+ 2179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.740628 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:20.740877 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:20.740914 (+ 37us) client_negotiation.cc:769] Sending connection context
0811 22:25:20.741031 (+ 117us) client_negotiation.cc:240] Negotiation successful
0811 22:25:20.741169 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":957,"mutex_wait_us":3,"thread_start_us":719,"threads_started":1}
I20260811 22:25:20.743012 32721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e16074c04f643e5b4a36d2ebf1cb3eb" candidate_uuid: "9664a73064234bc4912c8ddb99a59cb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f7acc1fba04ddebccc688885357742" is_pre_election: true
I20260811 22:25:20.743374 623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:20.731251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:37315 (local address 127.29.106.3:36999)
0811 22:25:20.731466 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:20.731471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:20.732785 (+ 1314us) server_negotiation.cc:407] Connection header received
0811 22:25:20.732995 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:20.733001 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:20.733078 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:20.733175 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:25:20.734719 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.736551 (+ 1832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.740223 (+ 3672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.741803 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.741966 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:20.742033 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:20.742803 (+ 770us) server_negotiation.cc:299] Negotiation successful
0811 22:25:20.742927 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260811 22:25:20.743790 32721 raft_consensus.cc:2454] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 85f7acc1fba04ddebccc688885357742 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9664a73064234bc4912c8ddb99a59cb0 in term 0.
I20260811 22:25:20.744278 722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:20.731186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.3:36999 (local address 127.29.106.1:37315)
0811 22:25:20.732668 (+ 1482us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:20.732686 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:20.732856 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:20.733210 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:20.733215 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:20.733240 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:25:20.734405 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:20.734421 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.736701 (+ 2280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.736707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:20.740029 (+ 3322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:20.740045 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.742358 (+ 2313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.742369 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:20.742611 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:20.742647 (+ 36us) client_negotiation.cc:769] Sending connection context
0811 22:25:20.743540 (+ 893us) client_negotiation.cc:240] Negotiation successful
0811 22:25:20.743671 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1289,"thread_start_us":897,"threads_started":1}
I20260811 22:25:20.744448 32347 leader_election.cc:302] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 9664a73064234bc4912c8ddb99a59cb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f7acc1fba04ddebccc688885357742, 9664a73064234bc4912c8ddb99a59cb0; no voters:
I20260811 22:25:20.745051 637 raft_consensus.cc:2781] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 9664a73064234bc4912c8ddb99a59cb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:25:20.745306 637 raft_consensus.cc:482] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 9664a73064234bc4912c8ddb99a59cb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:25:20.745509 637 raft_consensus.cc:3037] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 9664a73064234bc4912c8ddb99a59cb0 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:20.745679 501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e16074c04f643e5b4a36d2ebf1cb3eb" candidate_uuid: "9664a73064234bc4912c8ddb99a59cb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b139b7de16d4c39acd5ba4d9ab8648d" is_pre_election: true
I20260811 22:25:20.746011 501 raft_consensus.cc:2454] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 8b139b7de16d4c39acd5ba4d9ab8648d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9664a73064234bc4912c8ddb99a59cb0 in term 0.
I20260811 22:25:20.749378 637 raft_consensus.cc:504] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 9664a73064234bc4912c8ddb99a59cb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9664a73064234bc4912c8ddb99a59cb0" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43297 } } peers { permanent_uuid: "85f7acc1fba04ddebccc688885357742" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44367 } } peers { permanent_uuid: "8b139b7de16d4c39acd5ba4d9ab8648d" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36999 } }
I20260811 22:25:20.750111 637 leader_election.cc:288] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 9664a73064234bc4912c8ddb99a59cb0 [CANDIDATE]: Term 1 election: Requested vote from peers 85f7acc1fba04ddebccc688885357742 (127.29.106.2:44367), 8b139b7de16d4c39acd5ba4d9ab8648d (127.29.106.3:36999)
I20260811 22:25:20.750586 32721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e16074c04f643e5b4a36d2ebf1cb3eb" candidate_uuid: "9664a73064234bc4912c8ddb99a59cb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f7acc1fba04ddebccc688885357742"
I20260811 22:25:20.750759 501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e16074c04f643e5b4a36d2ebf1cb3eb" candidate_uuid: "9664a73064234bc4912c8ddb99a59cb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b139b7de16d4c39acd5ba4d9ab8648d"
I20260811 22:25:20.750895 32721 raft_consensus.cc:3037] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 85f7acc1fba04ddebccc688885357742 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:20.751027 501 raft_consensus.cc:3037] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 8b139b7de16d4c39acd5ba4d9ab8648d [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:20.754523 32721 raft_consensus.cc:2454] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 85f7acc1fba04ddebccc688885357742 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9664a73064234bc4912c8ddb99a59cb0 in term 1.
I20260811 22:25:20.754946 32347 leader_election.cc:302] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 9664a73064234bc4912c8ddb99a59cb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f7acc1fba04ddebccc688885357742, 9664a73064234bc4912c8ddb99a59cb0; no voters:
I20260811 22:25:20.755079 501 raft_consensus.cc:2454] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 8b139b7de16d4c39acd5ba4d9ab8648d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9664a73064234bc4912c8ddb99a59cb0 in term 1.
I20260811 22:25:20.755384 637 raft_consensus.cc:2781] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 9664a73064234bc4912c8ddb99a59cb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:25:20.756531 637 raft_consensus.cc:686] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 9664a73064234bc4912c8ddb99a59cb0 [term 1 LEADER]: Becoming Leader. State: Replica: 9664a73064234bc4912c8ddb99a59cb0, State: Running, Role: LEADER
I20260811 22:25:20.757091 637 consensus_queue.cc:237] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 9664a73064234bc4912c8ddb99a59cb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9664a73064234bc4912c8ddb99a59cb0" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43297 } } peers { permanent_uuid: "85f7acc1fba04ddebccc688885357742" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44367 } } peers { permanent_uuid: "8b139b7de16d4c39acd5ba4d9ab8648d" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36999 } }
I20260811 22:25:20.760221 723 tablet_replica.cc:439] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 9664a73064234bc4912c8ddb99a59cb0: TxnStatusTablet state changed. Reason: New leader 9664a73064234bc4912c8ddb99a59cb0. Latest consensus state: current_term: 1 leader_uuid: "9664a73064234bc4912c8ddb99a59cb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9664a73064234bc4912c8ddb99a59cb0" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43297 } } peers { permanent_uuid: "85f7acc1fba04ddebccc688885357742" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44367 } } peers { permanent_uuid: "8b139b7de16d4c39acd5ba4d9ab8648d" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36999 } } }
I20260811 22:25:20.760936 723 tablet_replica.cc:442] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 9664a73064234bc4912c8ddb99a59cb0: This TxnStatusTablet replica's current role is: LEADER
I20260811 22:25:20.761620 32178 catalog_manager.cc:5495] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 9664a73064234bc4912c8ddb99a59cb0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9664a73064234bc4912c8ddb99a59cb0 (127.29.106.1). New cstate: current_term: 1 leader_uuid: "9664a73064234bc4912c8ddb99a59cb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9664a73064234bc4912c8ddb99a59cb0" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85f7acc1fba04ddebccc688885357742" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b139b7de16d4c39acd5ba4d9ab8648d" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36999 } health_report { overall_health: UNKNOWN } } }
I20260811 22:25:20.764379 727 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260811 22:25:20.764829 727 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260811 22:25:20.783806 620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:20.776422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51116 (local address 127.29.106.1:43297)
0811 22:25:20.776617 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:20.776623 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:20.776855 (+ 232us) server_negotiation.cc:407] Connection header received
0811 22:25:20.777148 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:20.777154 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:20.777251 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:20.777403 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:25:20.777791 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.778943 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.780877 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.781607 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.781751 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:20.782288 (+ 537us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:25:20.783045 (+ 757us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:25:20.783199 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:20.783387 (+ 188us) server_negotiation.cc:299] Negotiation successful
0811 22:25:20.783496 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260811 22:25:20.783820 714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:20.776577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:43297 (local address 127.0.0.1:51116)
0811 22:25:20.776766 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:20.776785 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:20.776939 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:20.777433 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:20.777438 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:20.777451 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 22:25:20.777639 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:20.777650 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.779087 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.779093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:20.780696 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:20.780712 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.781778 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.781785 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:20.782012 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:20.782086 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 22:25:20.783179 (+ 1093us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 22:25:20.783204 (+ 25us) client_negotiation.cc:769] Sending connection context
0811 22:25:20.783347 (+ 143us) client_negotiation.cc:240] Negotiation successful
0811 22:25:20.783508 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260811 22:25:20.788877 501 raft_consensus.cc:1261] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 8b139b7de16d4c39acd5ba4d9ab8648d [term 1 FOLLOWER]: Refusing update from remote peer 9664a73064234bc4912c8ddb99a59cb0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:25:20.789201 32721 raft_consensus.cc:1261] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 85f7acc1fba04ddebccc688885357742 [term 1 FOLLOWER]: Refusing update from remote peer 9664a73064234bc4912c8ddb99a59cb0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:25:20.789666 637 consensus_queue.cc:1035] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 9664a73064234bc4912c8ddb99a59cb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b139b7de16d4c39acd5ba4d9ab8648d" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:25:20.789984 723 consensus_queue.cc:1035] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 9664a73064234bc4912c8ddb99a59cb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f7acc1fba04ddebccc688885357742" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:25:20.802871 667 tablet_replica.cc:439] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 85f7acc1fba04ddebccc688885357742: TxnStatusTablet state changed. Reason: New leader 9664a73064234bc4912c8ddb99a59cb0. Latest consensus state: current_term: 1 leader_uuid: "9664a73064234bc4912c8ddb99a59cb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9664a73064234bc4912c8ddb99a59cb0" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43297 } } peers { permanent_uuid: "85f7acc1fba04ddebccc688885357742" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44367 } } peers { permanent_uuid: "8b139b7de16d4c39acd5ba4d9ab8648d" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36999 } } }
I20260811 22:25:20.803185 667 tablet_replica.cc:442] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 85f7acc1fba04ddebccc688885357742: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 22:25:20.807780 637 tablet_replica.cc:439] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 9664a73064234bc4912c8ddb99a59cb0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9664a73064234bc4912c8ddb99a59cb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9664a73064234bc4912c8ddb99a59cb0" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43297 } } peers { permanent_uuid: "85f7acc1fba04ddebccc688885357742" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44367 } } peers { permanent_uuid: "8b139b7de16d4c39acd5ba4d9ab8648d" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36999 } } }
I20260811 22:25:20.808244 637 tablet_replica.cc:442] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 9664a73064234bc4912c8ddb99a59cb0: This TxnStatusTablet replica's current role is: LEADER
I20260811 22:25:20.810050 656 mvcc.cc:205] Tried to move back new op lower bound from 7317451246743707648 to 7317451246625755136. Current Snapshot: MvccSnapshot[applied={T|T < 7317451246743707648}]
I20260811 22:25:20.810612 667 tablet_replica.cc:439] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 85f7acc1fba04ddebccc688885357742: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9664a73064234bc4912c8ddb99a59cb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9664a73064234bc4912c8ddb99a59cb0" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43297 } } peers { permanent_uuid: "85f7acc1fba04ddebccc688885357742" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44367 } } peers { permanent_uuid: "8b139b7de16d4c39acd5ba4d9ab8648d" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36999 } } }
I20260811 22:25:20.810861 667 tablet_replica.cc:442] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 85f7acc1fba04ddebccc688885357742: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 22:25:20.815285 638 tablet_replica.cc:439] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 8b139b7de16d4c39acd5ba4d9ab8648d: TxnStatusTablet state changed. Reason: New leader 9664a73064234bc4912c8ddb99a59cb0. Latest consensus state: current_term: 1 leader_uuid: "9664a73064234bc4912c8ddb99a59cb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9664a73064234bc4912c8ddb99a59cb0" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43297 } } peers { permanent_uuid: "85f7acc1fba04ddebccc688885357742" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44367 } } peers { permanent_uuid: "8b139b7de16d4c39acd5ba4d9ab8648d" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36999 } } }
I20260811 22:25:20.815739 638 tablet_replica.cc:442] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 8b139b7de16d4c39acd5ba4d9ab8648d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 22:25:20.822759 723 tablet_replica.cc:439] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 9664a73064234bc4912c8ddb99a59cb0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9664a73064234bc4912c8ddb99a59cb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9664a73064234bc4912c8ddb99a59cb0" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43297 } } peers { permanent_uuid: "85f7acc1fba04ddebccc688885357742" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44367 } } peers { permanent_uuid: "8b139b7de16d4c39acd5ba4d9ab8648d" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36999 } } }
I20260811 22:25:20.823136 723 tablet_replica.cc:442] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 9664a73064234bc4912c8ddb99a59cb0: This TxnStatusTablet replica's current role is: LEADER
I20260811 22:25:20.825889 638 tablet_replica.cc:439] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 8b139b7de16d4c39acd5ba4d9ab8648d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9664a73064234bc4912c8ddb99a59cb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9664a73064234bc4912c8ddb99a59cb0" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43297 } } peers { permanent_uuid: "85f7acc1fba04ddebccc688885357742" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44367 } } peers { permanent_uuid: "8b139b7de16d4c39acd5ba4d9ab8648d" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36999 } } }
I20260811 22:25:20.826447 638 tablet_replica.cc:442] T 1e16074c04f643e5b4a36d2ebf1cb3eb P 8b139b7de16d4c39acd5ba4d9ab8648d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 22:25:20.827054 661 mvcc.cc:205] Tried to move back new op lower bound from 7317451246743707648 to 7317451246625755136. Current Snapshot: MvccSnapshot[applied={T|T < 7317451246743707648 or (T in {7317451246743707648})}]
I20260811 22:25:20.831588 623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:20.818715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39730 (local address 127.29.106.3:36999)
0811 22:25:20.818880 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:20.818885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:20.818905 (+ 20us) server_negotiation.cc:407] Connection header received
0811 22:25:20.818999 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:20.819004 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:20.819145 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:20.819317 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:25:20.820171 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.821708 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.824816 (+ 3108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.826004 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.826158 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:20.829622 (+ 3464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:25:20.830420 (+ 798us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:25:20.830576 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:20.831012 (+ 436us) server_negotiation.cc:299] Negotiation successful
0811 22:25:20.831100 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 22:25:20.861024 620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:20.848956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:33185 (local address 127.29.106.1:43297)
0811 22:25:20.849154 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:20.849160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:20.852199 (+ 3039us) server_negotiation.cc:407] Connection header received
0811 22:25:20.852427 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:20.852434 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:20.852526 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:20.852631 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:25:20.853753 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.855033 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.857904 (+ 2871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.859052 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.859200 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:20.859986 (+ 786us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:25:20.860180 (+ 194us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:25:20.860280 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:20.860461 (+ 181us) server_negotiation.cc:299] Negotiation successful
0811 22:25:20.860549 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260811 22:25:20.861133 741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:20.848538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:43297 (local address 127.29.106.3:33185)
0811 22:25:20.852074 (+ 3536us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:20.852095 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:20.852276 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:20.852664 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:20.852669 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:20.852681 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 22:25:20.853553 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:20.853570 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.855180 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.855185 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:20.857743 (+ 2558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:20.857760 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:20.859371 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:20.859381 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:20.859717 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:20.859796 (+ 79us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 22:25:20.860302 (+ 506us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 22:25:20.860314 (+ 12us) client_negotiation.cc:769] Sending connection context
0811 22:25:20.860403 (+ 89us) client_negotiation.cc:240] Negotiation successful
0811 22:25:20.860541 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3323,"thread_start_us":3099,"threads_started":1}
I20260811 22:25:20.873342 32466 raft_consensus.cc:1261] T a6125ea6a8754dc19cf98ff177382e05 P 9664a73064234bc4912c8ddb99a59cb0 [term 1 FOLLOWER]: Refusing update from remote peer 8b139b7de16d4c39acd5ba4d9ab8648d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:25:20.873898 32722 raft_consensus.cc:1261] T a6125ea6a8754dc19cf98ff177382e05 P 85f7acc1fba04ddebccc688885357742 [term 1 FOLLOWER]: Refusing update from remote peer 8b139b7de16d4c39acd5ba4d9ab8648d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:25:20.874296 696 consensus_queue.cc:1035] T a6125ea6a8754dc19cf98ff177382e05 P 8b139b7de16d4c39acd5ba4d9ab8648d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9664a73064234bc4912c8ddb99a59cb0" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260811 22:25:20.875202 638 consensus_queue.cc:1035] T a6125ea6a8754dc19cf98ff177382e05 P 8b139b7de16d4c39acd5ba4d9ab8648d [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f7acc1fba04ddebccc688885357742" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260811 22:25:20.987429 32450 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTabletLocations -> {remote=127.29.106.62:35421, user_credentials={real_user=slave}} blocked reactor thread for 48106.6us
I20260811 22:25:21.010196 623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:20.984529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:38655 (local address 127.29.106.3:36999)
0811 22:25:20.984692 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:20.984702 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:20.997661 (+ 12959us) server_negotiation.cc:407] Connection header received
0811 22:25:20.997954 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:20.997960 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:20.998035 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:20.998105 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:25:20.999685 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:21.001088 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:21.006539 (+ 5451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:21.007759 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:21.007867 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:21.008965 (+ 1098us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:25:21.009209 (+ 244us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:25:21.009286 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:21.009548 (+ 262us) server_negotiation.cc:299] Negotiation successful
0811 22:25:21.009637 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260811 22:25:21.011840 751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:20.988268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.3:36999 (local address 127.29.106.1:38655)
0811 22:25:20.997559 (+ 9291us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:20.997581 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:20.997751 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:20.998296 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:20.998301 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:20.998314 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 22:25:20.999476 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:20.999492 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:21.001252 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:21.001257 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:21.006388 (+ 5131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:21.006405 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:21.008322 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:21.008330 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:21.008599 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:21.008681 (+ 82us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 22:25:21.009390 (+ 709us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 22:25:21.009401 (+ 11us) client_negotiation.cc:769] Sending connection context
0811 22:25:21.010650 (+ 1249us) client_negotiation.cc:240] Negotiation successful
0811 22:25:21.010808 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8992,"thread_start_us":8729,"threads_started":1}
I20260811 22:25:21.073637 32178 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:25:21.074048 32178 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:25:21.078418 32178 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c2fed88d6bdc4913bd6dd4e7d89e1279: Sending DeleteTablet for 3 replicas of tablet a6125ea6a8754dc19cf98ff177382e05
I20260811 22:25:21.080231 32426 tablet_service.cc:1514] Processing DeleteTablet for tablet a6125ea6a8754dc19cf98ff177382e05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:25:21 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:51108
I20260811 22:25:21.081017 32684 tablet_service.cc:1514] Processing DeleteTablet for tablet a6125ea6a8754dc19cf98ff177382e05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:25:21 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:40674
I20260811 22:25:21.081853 462 tablet_service.cc:1514] Processing DeleteTablet for tablet a6125ea6a8754dc19cf98ff177382e05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:25:21 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:39716
I20260811 22:25:21.128579 760 tablet_replica.cc:330] T a6125ea6a8754dc19cf98ff177382e05 P 85f7acc1fba04ddebccc688885357742: stopping tablet replica
I20260811 22:25:21.128944 760 raft_consensus.cc:2229] T a6125ea6a8754dc19cf98ff177382e05 P 85f7acc1fba04ddebccc688885357742 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:25:21.130092 760 raft_consensus.cc:2258] T a6125ea6a8754dc19cf98ff177382e05 P 85f7acc1fba04ddebccc688885357742 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:25:21.131295 759 tablet_replica.cc:330] T a6125ea6a8754dc19cf98ff177382e05 P 9664a73064234bc4912c8ddb99a59cb0: stopping tablet replica
I20260811 22:25:21.131699 759 raft_consensus.cc:2229] T a6125ea6a8754dc19cf98ff177382e05 P 9664a73064234bc4912c8ddb99a59cb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:25:21.132263 761 tablet_replica.cc:330] T a6125ea6a8754dc19cf98ff177382e05 P 8b139b7de16d4c39acd5ba4d9ab8648d: stopping tablet replica
I20260811 22:25:21.132556 761 raft_consensus.cc:2229] T a6125ea6a8754dc19cf98ff177382e05 P 8b139b7de16d4c39acd5ba4d9ab8648d [term 1 LEADER]: Raft consensus shutting down.
I20260811 22:25:21.132887 759 raft_consensus.cc:2258] T a6125ea6a8754dc19cf98ff177382e05 P 9664a73064234bc4912c8ddb99a59cb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:25:21.133590 761 raft_consensus.cc:2258] T a6125ea6a8754dc19cf98ff177382e05 P 8b139b7de16d4c39acd5ba4d9ab8648d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:25:21.134441 759 ts_tablet_manager.cc:1905] T a6125ea6a8754dc19cf98ff177382e05 P 9664a73064234bc4912c8ddb99a59cb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:25:21.138962 760 ts_tablet_manager.cc:1905] T a6125ea6a8754dc19cf98ff177382e05 P 85f7acc1fba04ddebccc688885357742: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:25:21.143106 761 ts_tablet_manager.cc:1905] T a6125ea6a8754dc19cf98ff177382e05 P 8b139b7de16d4c39acd5ba4d9ab8648d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:25:21.164999 759 ts_tablet_manager.cc:1918] T a6125ea6a8754dc19cf98ff177382e05 P 9664a73064234bc4912c8ddb99a59cb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 22:25:21.165170 761 ts_tablet_manager.cc:1918] T a6125ea6a8754dc19cf98ff177382e05 P 8b139b7de16d4c39acd5ba4d9ab8648d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 22:25:21.165422 761 log.cc:1196] T a6125ea6a8754dc19cf98ff177382e05 P 8b139b7de16d4c39acd5ba4d9ab8648d: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-2/wal/wals/a6125ea6a8754dc19cf98ff177382e05
I20260811 22:25:21.166185 761 ts_tablet_manager.cc:1939] T a6125ea6a8754dc19cf98ff177382e05 P 8b139b7de16d4c39acd5ba4d9ab8648d: Deleting consensus metadata
I20260811 22:25:21.167073 760 ts_tablet_manager.cc:1918] T a6125ea6a8754dc19cf98ff177382e05 P 85f7acc1fba04ddebccc688885357742: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 22:25:21.167320 760 log.cc:1196] T a6125ea6a8754dc19cf98ff177382e05 P 85f7acc1fba04ddebccc688885357742: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-1/wal/wals/a6125ea6a8754dc19cf98ff177382e05
I20260811 22:25:21.167542 32159 catalog_manager.cc:4841] TS 8b139b7de16d4c39acd5ba4d9ab8648d (127.29.106.3:36999): tablet a6125ea6a8754dc19cf98ff177382e05 (table test-table [id=2fcb37bd8b0542339d95edff45b9add5]) successfully deleted
I20260811 22:25:21.169063 759 log.cc:1196] T a6125ea6a8754dc19cf98ff177382e05 P 9664a73064234bc4912c8ddb99a59cb0: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786487105757681-30120-0/minicluster-data/ts-0/wal/wals/a6125ea6a8754dc19cf98ff177382e05
I20260811 22:25:21.169584 759 ts_tablet_manager.cc:1939] T a6125ea6a8754dc19cf98ff177382e05 P 9664a73064234bc4912c8ddb99a59cb0: Deleting consensus metadata
I20260811 22:25:21.170907 32158 catalog_manager.cc:4841] TS 9664a73064234bc4912c8ddb99a59cb0 (127.29.106.1:43297): tablet a6125ea6a8754dc19cf98ff177382e05 (table test-table [id=2fcb37bd8b0542339d95edff45b9add5]) successfully deleted
I20260811 22:25:21.175253 760 ts_tablet_manager.cc:1939] T a6125ea6a8754dc19cf98ff177382e05 P 85f7acc1fba04ddebccc688885357742: Deleting consensus metadata
I20260811 22:25:21.176656 32159 catalog_manager.cc:4841] TS 85f7acc1fba04ddebccc688885357742 (127.29.106.2:44367): tablet a6125ea6a8754dc19cf98ff177382e05 (table test-table [id=2fcb37bd8b0542339d95edff45b9add5]) successfully deleted
Aug 11 22:25:21 dist-test-slave-gjrz krb5kdc[32108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487120, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.106.1@KRBTEST.COM
I20260811 22:25:21.198342 620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:21.131984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51124 (local address 127.29.106.1:43297)
0811 22:25:21.132177 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:21.132184 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:21.166913 (+ 34729us) server_negotiation.cc:407] Connection header received
0811 22:25:21.167520 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:21.167528 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:21.167624 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:21.167754 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:21.172216 (+ 4462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:21.173859 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:21.177658 (+ 3799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:21.178742 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:21.178917 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:21.187491 (+ 8574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:21.187513 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:21.187536 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:21.187619 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:21.192906 (+ 5287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:21.193964 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:21.193975 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:21.193980 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:21.194235 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:21.194780 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:21.194785 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:21.194788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:21.195440 (+ 652us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:21.196019 (+ 579us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:21.197322 (+ 1303us) server_negotiation.cc:299] Negotiation successful
0811 22:25:21.197559 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260811 22:25:21.200299 32556 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51124
I20260811 22:25:21.258539 766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:21.214325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41570 (local address 127.29.106.62:35421)
0811 22:25:21.232845 (+ 18520us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:21.232852 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:21.232876 (+ 24us) server_negotiation.cc:407] Connection header received
0811 22:25:21.232945 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:21.232950 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:21.233033 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:21.233144 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:21.236736 (+ 3592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:21.238669 (+ 1933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:21.241121 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:21.242137 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:21.242251 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:21.247292 (+ 5041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:21.247300 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:21.247306 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:21.247382 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:21.253246 (+ 5864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:21.254441 (+ 1195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:21.254447 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:21.254451 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:21.254712 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:21.255343 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:21.255349 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:21.255352 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:21.255819 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:21.256198 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:21.257686 (+ 1488us) server_negotiation.cc:299] Negotiation successful
0811 22:25:21.258000 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18318,"thread_start_us":22242,"threads_started":1}
W20260811 22:25:21.259524 32178 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41570
I20260811 22:25:21.262823 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 32315
I20260811 22:25:21.297078 620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:21.265910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51134 (local address 127.29.106.1:43297)
0811 22:25:21.266103 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:21.266109 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:21.274835 (+ 8726us) server_negotiation.cc:407] Connection header received
0811 22:25:21.275235 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:21.275241 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:21.275345 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:21.275441 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:21.277546 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:21.278979 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:21.281135 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:21.282493 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:21.282602 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:21.287493 (+ 4891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:21.287499 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:21.287503 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:21.287544 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:21.292951 (+ 5407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:21.293912 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:21.293918 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:21.293921 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:21.294206 (+ 285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:21.294792 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:21.294797 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:21.294800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:21.295226 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:21.295557 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:21.296315 (+ 758us) server_negotiation.cc:299] Negotiation successful
0811 22:25:21.296623 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260811 22:25:21.298949 32556 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51134
W20260811 22:25:21.299654 30120 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 32315: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 22:25:21.299885 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 32315
I20260811 22:25:21.333256 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 32574
Aug 11 22:25:21 dist-test-slave-gjrz krb5kdc[32108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487120, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.106.2@KRBTEST.COM
I20260811 22:25:21.351061 773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:21.334033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40694 (local address 127.29.106.2:44367)
0811 22:25:21.334546 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:21.334551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:21.334566 (+ 15us) server_negotiation.cc:407] Connection header received
0811 22:25:21.334620 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:21.334624 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:21.334682 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:21.334773 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:21.336332 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:21.337118 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:21.339126 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:21.339592 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:21.339688 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:21.345084 (+ 5396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:21.345098 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:21.345114 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:21.345147 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:21.348372 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:21.349090 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:21.349098 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:21.349100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:21.349237 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:21.349600 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:21.349604 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:21.349606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:21.349935 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:21.350278 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:21.350503 (+ 225us) server_negotiation.cc:299] Negotiation successful
0811 22:25:21.350719 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":285,"threads_started":1}
W20260811 22:25:21.351748 339 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40694
W20260811 22:25:21.352357 30120 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 32574: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 22:25:21.352509 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 32574
I20260811 22:25:21.386145 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 355
Aug 11 22:25:21 dist-test-slave-gjrz krb5kdc[32108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487120, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.106.3@KRBTEST.COM
I20260811 22:25:21.402269 623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:21.386938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39736 (local address 127.29.106.3:36999)
0811 22:25:21.387120 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:21.387125 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:21.387351 (+ 226us) server_negotiation.cc:407] Connection header received
0811 22:25:21.387547 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:21.387553 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:21.387633 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:21.387717 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:21.388952 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:21.389691 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:21.390885 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:21.391386 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:21.391500 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:21.395402 (+ 3902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:21.395417 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:21.395434 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:21.395474 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:21.399501 (+ 4027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:21.400143 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:21.400152 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:21.400160 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:21.400324 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:21.400706 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:21.400711 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:21.400715 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:21.401174 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:21.401490 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:21.401646 (+ 156us) server_negotiation.cc:299] Negotiation successful
0811 22:25:21.401867 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260811 22:25:21.402937 585 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39736
W20260811 22:25:21.403584 30120 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 355: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 22:25:21.403741 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 355
I20260811 22:25:21.431689 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 32132
I20260811 22:25:21.432786 32258 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:25:21.549427 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 32132
2026-08-11T22:25:21Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5032 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:25:21 dist-test-slave-gjrz krb5kdc[778](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:25:21 dist-test-slave-gjrz krb5kdc[778](info): set up 2 sockets
Aug 11 22:25:21 dist-test-slave-gjrz krb5kdc[778](info): commencing operation
krb5kdc: starting...
W20260811 22:25:22.809419 30120 mini_kdc.cc:121] Time spent starting KDC: real 1.209s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Aug 11 22:25:22 dist-test-slave-gjrz krb5kdc[778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:25:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:25:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.106.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
I20260811 22:25:22.986155 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:40297
--webserver_interface=127.29.106.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:41861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:40297
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:25:23.300181 813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:23.300428 813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:23.300515 813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:23.307508 813 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:25:23.307598 813 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260811 22:25:23.307641 813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:23.307704 813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:23.307753 813 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:25:23.307804 813 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:25:23.317108 813 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:41861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:40297
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:40297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.106.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:25:23.318190 813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:23.319450 813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:23.330685 821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:23.330757 826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:23.330734 823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:23.331569 813 server_base.cc:1034] running on GCE node
I20260811 22:25:23.332275 813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:23.333513 813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:23.334750 813 hybrid_clock.cc:648] HybridClock initialized: now 1786487123334677 us; error 41 us; skew 500 ppm
I20260811 22:25:23.335094 813 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:25:23 dist-test-slave-gjrz krb5kdc[778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487123, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:23.338088 813 init.cc:377] Logged in from keytab as kudu/127.29.106.62@KRBTEST.COM (short username kudu)
I20260811 22:25:23.339532 813 webserver.cc:460] Webserver started at http://127.29.106.62:46625/ using document root <none> and password file <none>
I20260811 22:25:23.340181 813 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:23.340283 813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:23.340571 813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:23.343418 813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "fc4020c4fbb7446aadc65b89aeb691ba"
format_stamp: "Formatted at 2026-08-11 22:25:23 on dist-test-slave-gjrz"
server_key: "c2c57e8f97176d30964f189422693278"
server_key_iv: "e4dd6a3a669c319a1ccf50282d2a01b2"
server_key_version: "encryptionkey@0"
I20260811 22:25:23.344162 813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "fc4020c4fbb7446aadc65b89aeb691ba"
format_stamp: "Formatted at 2026-08-11 22:25:23 on dist-test-slave-gjrz"
server_key: "c2c57e8f97176d30964f189422693278"
server_key_iv: "e4dd6a3a669c319a1ccf50282d2a01b2"
server_key_version: "encryptionkey@0"
I20260811 22:25:23.349071 813 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260811 22:25:23.352893 843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:23.354094 813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260811 22:25:23.354241 813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "fc4020c4fbb7446aadc65b89aeb691ba"
format_stamp: "Formatted at 2026-08-11 22:25:23 on dist-test-slave-gjrz"
server_key: "c2c57e8f97176d30964f189422693278"
server_key_iv: "e4dd6a3a669c319a1ccf50282d2a01b2"
server_key_version: "encryptionkey@0"
I20260811 22:25:23.354429 813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:23.382366 813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:23.386045 813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:23.386293 813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:23.405767 813 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.62:40297
I20260811 22:25:23.405774 952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.62:40297 every 8 connection(s)
I20260811 22:25:23.407006 813 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:25:23.410131 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 813
I20260811 22:25:23.410302 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:25:23.410569 953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:23.410938 30120 external_mini_cluster.cc:1428] Setting key e8ef54a5bd3d471abc6532be08431852
I20260811 22:25:23.424413 953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc4020c4fbb7446aadc65b89aeb691ba: Bootstrap starting.
Aug 11 22:25:23 dist-test-slave-gjrz krb5kdc[778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:25:23.427997 953 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc4020c4fbb7446aadc65b89aeb691ba: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:23.429339 953 log.cc:824] T 00000000000000000000000000000000 P fc4020c4fbb7446aadc65b89aeb691ba: Log is configured to *not* fsync() on all Append() calls
I20260811 22:25:23.432689 953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc4020c4fbb7446aadc65b89aeb691ba: No bootstrap required, opened a new log
I20260811 22:25:23.434265 962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:23.412805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40124 (local address 127.29.106.62:40297)
0811 22:25:23.413666 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:23.413680 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:23.413734 (+ 54us) server_negotiation.cc:407] Connection header received
0811 22:25:23.415552 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:23.415585 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:23.416065 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:23.416492 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:23.417797 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:23.419707 (+ 1910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:23.420982 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:23.422060 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:23.422255 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:23.425995 (+ 3740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:23.426005 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:23.426024 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:23.426083 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:23.429820 (+ 3737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:23.430657 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:23.430664 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:23.430670 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:23.430878 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:23.431242 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:23.431247 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:23.431250 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:23.431913 (+ 663us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:23.432216 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:23.432569 (+ 353us) server_negotiation.cc:299] Negotiation successful
0811 22:25:23.432972 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":371,"threads_started":1}
I20260811 22:25:23.440908 953 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc4020c4fbb7446aadc65b89aeb691ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc4020c4fbb7446aadc65b89aeb691ba" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 40297 } }
I20260811 22:25:23.441216 953 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc4020c4fbb7446aadc65b89aeb691ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:25:23.441285 953 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc4020c4fbb7446aadc65b89aeb691ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc4020c4fbb7446aadc65b89aeb691ba, State: Initialized, Role: FOLLOWER
I20260811 22:25:23.441776 953 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc4020c4fbb7446aadc65b89aeb691ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc4020c4fbb7446aadc65b89aeb691ba" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 40297 } }
I20260811 22:25:23.442003 953 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc4020c4fbb7446aadc65b89aeb691ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:25:23.442103 953 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc4020c4fbb7446aadc65b89aeb691ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:25:23.442219 953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc4020c4fbb7446aadc65b89aeb691ba [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:23.444412 953 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc4020c4fbb7446aadc65b89aeb691ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc4020c4fbb7446aadc65b89aeb691ba" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 40297 } }
I20260811 22:25:23.444828 953 leader_election.cc:302] T 00000000000000000000000000000000 P fc4020c4fbb7446aadc65b89aeb691ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc4020c4fbb7446aadc65b89aeb691ba; no voters:
I20260811 22:25:23.445313 953 leader_election.cc:288] T 00000000000000000000000000000000 P fc4020c4fbb7446aadc65b89aeb691ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:25:23.445436 965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc4020c4fbb7446aadc65b89aeb691ba [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:25:23.446089 965 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc4020c4fbb7446aadc65b89aeb691ba [term 1 LEADER]: Becoming Leader. State: Replica: fc4020c4fbb7446aadc65b89aeb691ba, State: Running, Role: LEADER
I20260811 22:25:23.446578 965 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc4020c4fbb7446aadc65b89aeb691ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc4020c4fbb7446aadc65b89aeb691ba" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 40297 } }
I20260811 22:25:23.447190 953 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc4020c4fbb7446aadc65b89aeb691ba [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:25:23.449219 967 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc4020c4fbb7446aadc65b89aeb691ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc4020c4fbb7446aadc65b89aeb691ba. Latest consensus state: current_term: 1 leader_uuid: "fc4020c4fbb7446aadc65b89aeb691ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc4020c4fbb7446aadc65b89aeb691ba" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 40297 } } }
I20260811 22:25:23.449687 967 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc4020c4fbb7446aadc65b89aeb691ba [sys.catalog]: This master's current role is: LEADER
I20260811 22:25:23.449292 966 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc4020c4fbb7446aadc65b89aeb691ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc4020c4fbb7446aadc65b89aeb691ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc4020c4fbb7446aadc65b89aeb691ba" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 40297 } } }
I20260811 22:25:23.455374 966 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc4020c4fbb7446aadc65b89aeb691ba [sys.catalog]: This master's current role is: LEADER
I20260811 22:25:23.455925 973 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:25:23.460732 973 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:25:23.468941 973 catalog_manager.cc:1347] Generated new cluster ID: 20c17767e78542738638e9bf71e1cd89
I20260811 22:25:23.469048 973 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:25:23.503248 973 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:25:23.504491 973 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:25:23.517155 973 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc4020c4fbb7446aadc65b89aeb691ba: Generated new TSK 0
I20260811 22:25:23.517942 973 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.106.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab.
Entry for principal kudu/127.29.106.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab.
I20260811 22:25:23.567420 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.1:0
--local_ip_for_outbound_sockets=127.29.106.1
--webserver_interface=127.29.106.1
--webserver_port=0
--tserver_master_addrs=127.29.106.62:40297
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab
--principal=kudu/127.29.106.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:41861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:25:23.878094 1001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:23.878341 1001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:23.878481 1001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:23.885381 1001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:23.885474 1001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:23.885704 1001 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260811 22:25:23.885804 1001 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 22:25:23.885908 1001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.1
I20260811 22:25:23.895349 1001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:41861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab
--principal=kudu/127.29.106.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.106.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.106.62:40297
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.1
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:25:23.896487 1001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:23.897750 1001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:23.909868 1016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:23.909900 1019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:23.909924 1021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:23.910642 1001 server_base.cc:1034] running on GCE node
I20260811 22:25:23.911196 1001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:23.912068 1001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:23.913275 1001 hybrid_clock.cc:648] HybridClock initialized: now 1786487123913229 us; error 33 us; skew 500 ppm
I20260811 22:25:23.913563 1001 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:25:23 dist-test-slave-gjrz krb5kdc[778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487123, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:23.916548 1001 init.cc:377] Logged in from keytab as kudu/127.29.106.1@KRBTEST.COM (short username kudu)
I20260811 22:25:23.917954 1001 webserver.cc:460] Webserver started at http://127.29.106.1:34355/ using document root <none> and password file <none>
I20260811 22:25:23.918565 1001 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:23.918681 1001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:23.918963 1001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:23.921727 1001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-0/data/instance:
uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5"
format_stamp: "Formatted at 2026-08-11 22:25:23 on dist-test-slave-gjrz"
server_key: "fadd636cfdfe53ccb99e738f8c13d90d"
server_key_iv: "1eb76d75b08de19b549e122dd74b337f"
server_key_version: "encryptionkey@0"
I20260811 22:25:23.922458 1001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance:
uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5"
format_stamp: "Formatted at 2026-08-11 22:25:23 on dist-test-slave-gjrz"
server_key: "fadd636cfdfe53ccb99e738f8c13d90d"
server_key_iv: "1eb76d75b08de19b549e122dd74b337f"
server_key_version: "encryptionkey@0"
I20260811 22:25:23.927232 1001 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260811 22:25:23.930405 1043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:23.931710 1001 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:25:23.931854 1001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-0/wal
uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5"
format_stamp: "Formatted at 2026-08-11 22:25:23 on dist-test-slave-gjrz"
server_key: "fadd636cfdfe53ccb99e738f8c13d90d"
server_key_iv: "1eb76d75b08de19b549e122dd74b337f"
server_key_version: "encryptionkey@0"
I20260811 22:25:23.932018 1001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:23.955410 1001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:23.959352 1001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:23.959607 1001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:23.962087 1001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:25:23.962193 1001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:23.962297 1001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:25:23.962426 1001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Aug 11 22:25:23 dist-test-slave-gjrz krb5kdc[778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487123, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.1@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:25:23.982946 1059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:23.962537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:47857 (local address 127.29.106.62:40297)
0811 22:25:23.963629 (+ 1092us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:23.963635 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:23.964345 (+ 710us) server_negotiation.cc:407] Connection header received
0811 22:25:23.966299 (+ 1954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:23.966304 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:23.966389 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:23.966482 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:23.968583 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:23.969709 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:23.972084 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:23.972631 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:23.972745 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:23.976854 (+ 4109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:23.976861 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:23.976866 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:23.976908 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:23.979747 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:23.980635 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:23.980641 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:23.980644 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:23.980800 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:23.981270 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:23.981275 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:23.981279 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:23.981557 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:23.981688 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:23.982371 (+ 683us) server_negotiation.cc:299] Negotiation successful
0811 22:25:23.982583 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":959,"thread_start_us":845,"threads_started":1}
I20260811 22:25:23.983850 1060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:23.963207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:40297 (local address 127.29.106.1:47857)
0811 22:25:23.964171 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:23.964218 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:23.966020 (+ 1802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:23.966748 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:23.966760 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:23.967534 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:23.968337 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:23.968372 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:23.969873 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:23.969878 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:23.971939 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:23.971952 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:23.972878 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:23.972886 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:23.973089 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:23.973698 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:23.973719 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:23.976591 (+ 2872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:23.979861 (+ 3270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:23.979870 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:23.979884 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:23.980472 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:23.980947 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:23.980950 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:23.980952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:23.981152 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:23.981677 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:23.981684 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:23.982014 (+ 330us) client_negotiation.cc:769] Sending connection context
0811 22:25:23.982244 (+ 230us) client_negotiation.cc:240] Negotiation successful
0811 22:25:23.982706 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":387,"threads_started":1}
I20260811 22:25:24.001497 1059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:23.989361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:43259 (local address 127.29.106.62:40297)
0811 22:25:23.989538 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:23.989544 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:23.990047 (+ 503us) server_negotiation.cc:407] Connection header received
0811 22:25:23.990171 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:23.990177 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:23.990254 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:23.990358 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:23.991224 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:23.992286 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:23.993517 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:23.994141 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:23.994263 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:23.996001 (+ 1738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:23.996005 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:23.996009 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:23.996038 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:23.998777 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:23.999572 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:23.999577 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:23.999581 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:23.999704 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:24.000041 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:24.000046 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:24.000050 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:24.000311 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:24.000435 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:24.001000 (+ 565us) server_negotiation.cc:299] Negotiation successful
0811 22:25:24.001201 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260811 22:25:24.002282 1188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:23.989378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:40297 (local address 127.29.106.1:43259)
0811 22:25:23.989966 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:23.989984 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:23.990087 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:23.990368 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:23.990370 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:23.990751 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:23.991101 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:23.991111 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:23.992404 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:23.992408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:23.993389 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:23.993401 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:23.994320 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:23.994324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:23.994440 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:23.994955 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:23.994970 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:23.995786 (+ 816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:23.998901 (+ 3115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:23.998906 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:23.998910 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:23.999460 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:23.999809 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:23.999814 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:23.999818 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:23.999944 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:24.000423 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:24.000431 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:24.000698 (+ 267us) client_negotiation.cc:769] Sending connection context
0811 22:25:24.001715 (+ 1017us) client_negotiation.cc:240] Negotiation successful
0811 22:25:24.001983 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":316,"threads_started":1}
I20260811 22:25:24.012477 1001 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.1:33721
I20260811 22:25:24.012509 1274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.1:33721 every 8 connection(s)
I20260811 22:25:24.013598 1001 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
I20260811 22:25:24.015323 1277 heartbeater.cc:347] Connected to a master server at 127.29.106.62:40297
I20260811 22:25:24.015517 1277 heartbeater.cc:464] Registering TS with master...
I20260811 22:25:24.016088 1277 heartbeater.cc:511] Master 127.29.106.62:40297 requested a full tablet report, sending...
I20260811 22:25:24.018019 877 ts_manager.cc:194] Registered new tserver with Master: 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 (127.29.106.1:33721)
I20260811 22:25:24.019496 877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.1@KRBTEST.COM'} at 127.29.106.1:47857
I20260811 22:25:24.023880 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 1001
I20260811 22:25:24.024058 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance
I20260811 22:25:24.024641 30120 external_mini_cluster.cc:1428] Setting key d0f74946d7d479e693b459a5a639f327
WARNING: no policy specified for kudu/127.29.106.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab.
Entry for principal kudu/127.29.106.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab.
I20260811 22:25:24.063558 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.2:0
--local_ip_for_outbound_sockets=127.29.106.2
--webserver_interface=127.29.106.2
--webserver_port=0
--tserver_master_addrs=127.29.106.62:40297
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab
--principal=kudu/127.29.106.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:41861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:25:24.373169 1289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:24.373416 1289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:24.373554 1289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:24.380527 1289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:24.380621 1289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:24.380741 1289 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260811 22:25:24.380803 1289 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 22:25:24.380899 1289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.2
I20260811 22:25:24.390439 1289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:41861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab
--principal=kudu/127.29.106.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.106.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.106.62:40297
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.2
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:25:24.391640 1289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:24.392872 1289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:24.404863 1304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:24.404826 1307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:24.405082 1310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:24.406694 1289 server_base.cc:1034] running on GCE node
I20260811 22:25:24.407299 1289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:24.408164 1289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:24.409366 1289 hybrid_clock.cc:648] HybridClock initialized: now 1786487124409318 us; error 39 us; skew 500 ppm
I20260811 22:25:24.409677 1289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:25:24 dist-test-slave-gjrz krb5kdc[778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487124, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:24.412783 1289 init.cc:377] Logged in from keytab as kudu/127.29.106.2@KRBTEST.COM (short username kudu)
I20260811 22:25:24.414139 1289 webserver.cc:460] Webserver started at http://127.29.106.2:46279/ using document root <none> and password file <none>
I20260811 22:25:24.414799 1289 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:24.414913 1289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:24.415192 1289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:24.417979 1289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-1/data/instance:
uuid: "a669f1e0074e4968bc0853b56071bdc0"
format_stamp: "Formatted at 2026-08-11 22:25:24 on dist-test-slave-gjrz"
server_key: "211ddb37a19e12fe820e0ff369481a62"
server_key_iv: "a44582ba5842416e50ef9c2531d06e46"
server_key_version: "encryptionkey@0"
I20260811 22:25:24.418699 1289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance:
uuid: "a669f1e0074e4968bc0853b56071bdc0"
format_stamp: "Formatted at 2026-08-11 22:25:24 on dist-test-slave-gjrz"
server_key: "211ddb37a19e12fe820e0ff369481a62"
server_key_iv: "a44582ba5842416e50ef9c2531d06e46"
server_key_version: "encryptionkey@0"
I20260811 22:25:24.423678 1289 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260811 22:25:24.426991 1327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:24.428236 1289 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:25:24.428385 1289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-1/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-1/wal
uuid: "a669f1e0074e4968bc0853b56071bdc0"
format_stamp: "Formatted at 2026-08-11 22:25:24 on dist-test-slave-gjrz"
server_key: "211ddb37a19e12fe820e0ff369481a62"
server_key_iv: "a44582ba5842416e50ef9c2531d06e46"
server_key_version: "encryptionkey@0"
I20260811 22:25:24.428572 1289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:24.465842 1289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:24.469502 1289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:24.469775 1289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:24.472332 1289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:25:24.472445 1289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:24.472622 1289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:25:24.472711 1289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 22:25:24 dist-test-slave-gjrz krb5kdc[778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487124, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.2@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:25:24.494871 1059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:24.472740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:59609 (local address 127.29.106.62:40297)
0811 22:25:24.472951 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:24.472959 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:24.474439 (+ 1480us) server_negotiation.cc:407] Connection header received
0811 22:25:24.477200 (+ 2761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:24.477207 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:24.477294 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:24.477404 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:24.479776 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:24.480682 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:24.483488 (+ 2806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:24.483910 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:24.484006 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:24.488871 (+ 4865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:24.488875 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:24.488879 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:24.488912 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:24.491939 (+ 3027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:24.492698 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:24.492704 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:24.492707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:24.492860 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:24.493259 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:24.493265 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:24.493268 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:24.493539 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:24.493682 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:24.494321 (+ 639us) server_negotiation.cc:299] Negotiation successful
0811 22:25:24.494539 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260811 22:25:24.496007 1341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:24.473356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:40297 (local address 127.29.106.2:59609)
0811 22:25:24.474253 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:24.474303 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:24.476843 (+ 2540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:24.477671 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:24.477681 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:24.478505 (+ 824us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:24.479506 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:24.479537 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:24.480862 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:24.480867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:24.483334 (+ 2467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:24.483350 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:24.484140 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:24.484147 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:24.484474 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:24.485271 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:24.485308 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:24.488645 (+ 3337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:24.492075 (+ 3430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:24.492083 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:24.492102 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:24.492572 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:24.492960 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:24.492965 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:24.492969 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:24.493148 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:24.493670 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:24.493678 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:24.493985 (+ 307us) client_negotiation.cc:769] Sending connection context
0811 22:25:24.494202 (+ 217us) client_negotiation.cc:240] Negotiation successful
0811 22:25:24.494677 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":367,"threads_started":1}
I20260811 22:25:24.514674 1453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:24.501149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:40297 (local address 127.29.106.2:38637)
0811 22:25:24.501773 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:24.501792 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:24.501940 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:24.502223 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:24.502228 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:24.502780 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:24.503167 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:24.503179 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:24.504214 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:24.504219 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:24.505735 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:24.505747 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:24.506750 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:24.506755 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:24.506908 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:24.507599 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:24.507624 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:24.508563 (+ 939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:24.512080 (+ 3517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:24.512086 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:24.512090 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:24.512587 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:24.513032 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:24.513037 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:24.513042 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:24.513210 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:24.513895 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:24.513898 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:24.514044 (+ 146us) client_negotiation.cc:769] Sending connection context
0811 22:25:24.514155 (+ 111us) client_negotiation.cc:240] Negotiation successful
0811 22:25:24.514367 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":302,"threads_started":1}
I20260811 22:25:24.514845 1059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:24.501149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:38637 (local address 127.29.106.62:40297)
0811 22:25:24.501429 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:24.501433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:24.501882 (+ 449us) server_negotiation.cc:407] Connection header received
0811 22:25:24.502057 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:24.502061 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:24.502115 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:24.502181 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:24.503333 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:24.504040 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:24.505860 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:24.506396 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:24.506511 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:24.508766 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:24.508773 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:24.508779 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:24.508821 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:24.511932 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:24.512750 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:24.512756 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:24.512759 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:24.512918 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:24.513334 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:24.513340 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:24.513343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:24.513779 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:24.513938 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:24.514261 (+ 323us) server_negotiation.cc:299] Negotiation successful
0811 22:25:24.514504 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189}
I20260811 22:25:24.525520 1289 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.2:37479
I20260811 22:25:24.525563 1564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.2:37479 every 8 connection(s)
I20260811 22:25:24.526643 1289 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
I20260811 22:25:24.528589 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 1289
I20260811 22:25:24.528614 1567 heartbeater.cc:347] Connected to a master server at 127.29.106.62:40297
I20260811 22:25:24.528757 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance
I20260811 22:25:24.528802 1567 heartbeater.cc:464] Registering TS with master...
I20260811 22:25:24.529193 30120 external_mini_cluster.cc:1428] Setting key 0b37f11d8bb438d4a82425d943623048
I20260811 22:25:24.529431 1567 heartbeater.cc:511] Master 127.29.106.62:40297 requested a full tablet report, sending...
I20260811 22:25:24.530683 876 ts_manager.cc:194] Registered new tserver with Master: a669f1e0074e4968bc0853b56071bdc0 (127.29.106.2:37479)
I20260811 22:25:24.531560 876 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.2@KRBTEST.COM'} at 127.29.106.2:59609
WARNING: no policy specified for kudu/127.29.106.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab.
Entry for principal kudu/127.29.106.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab.
I20260811 22:25:24.565757 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.3:0
--local_ip_for_outbound_sockets=127.29.106.3
--webserver_interface=127.29.106.3
--webserver_port=0
--tserver_master_addrs=127.29.106.62:40297
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab
--principal=kudu/127.29.106.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:41861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:25:24.883545 1574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:24.883898 1574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:24.884080 1574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:24.894865 1574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:24.894976 1574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:24.895115 1574 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260811 22:25:24.895182 1574 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 22:25:24.895303 1574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.3
I20260811 22:25:24.904840 1574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:41861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab
--principal=kudu/127.29.106.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.106.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.106.62:40297
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.3
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:25:24.905977 1574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:24.907349 1574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:24.919778 1582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:24.919859 1581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:24.919898 1585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:24.920523 1574 server_base.cc:1034] running on GCE node
I20260811 22:25:24.921105 1574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:24.921947 1574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:24.923161 1574 hybrid_clock.cc:648] HybridClock initialized: now 1786487124923113 us; error 43 us; skew 500 ppm
I20260811 22:25:24.923468 1574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:25:24 dist-test-slave-gjrz krb5kdc[778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487124, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:24.926441 1574 init.cc:377] Logged in from keytab as kudu/127.29.106.3@KRBTEST.COM (short username kudu)
I20260811 22:25:24.927762 1574 webserver.cc:460] Webserver started at http://127.29.106.3:33095/ using document root <none> and password file <none>
I20260811 22:25:24.928377 1574 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:24.928489 1574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:24.928773 1574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:24.931519 1574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-2/data/instance:
uuid: "eb362cc342004cdc9d75207b00cbee35"
format_stamp: "Formatted at 2026-08-11 22:25:24 on dist-test-slave-gjrz"
server_key: "e96ea58374118146619786e072e41643"
server_key_iv: "ac7f6e80843e583284e0881516984f2d"
server_key_version: "encryptionkey@0"
I20260811 22:25:24.932220 1574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance:
uuid: "eb362cc342004cdc9d75207b00cbee35"
format_stamp: "Formatted at 2026-08-11 22:25:24 on dist-test-slave-gjrz"
server_key: "e96ea58374118146619786e072e41643"
server_key_iv: "ac7f6e80843e583284e0881516984f2d"
server_key_version: "encryptionkey@0"
I20260811 22:25:24.936957 1574 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 22:25:24.940276 1604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:24.941522 1574 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 22:25:24.941658 1574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-2/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-2/wal
uuid: "eb362cc342004cdc9d75207b00cbee35"
format_stamp: "Formatted at 2026-08-11 22:25:24 on dist-test-slave-gjrz"
server_key: "e96ea58374118146619786e072e41643"
server_key_iv: "ac7f6e80843e583284e0881516984f2d"
server_key_version: "encryptionkey@0"
I20260811 22:25:24.941828 1574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:24.959446 1574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:24.963073 1574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:24.963346 1574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:24.965749 1574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:25:24.965845 1574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:24.965950 1574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:25:24.966022 1574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 22:25:24 dist-test-slave-gjrz krb5kdc[778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487124, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.3@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:25:24.989321 1627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:24.966870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:40297 (local address 127.29.106.3:53435)
0811 22:25:24.967745 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:24.967793 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:24.969737 (+ 1944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:24.970527 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:24.970549 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:24.971386 (+ 837us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:24.972371 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:24.972404 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:24.973567 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:24.973573 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:24.975591 (+ 2018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:24.975607 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:24.976731 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:24.976737 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:24.976974 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:24.977673 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:24.977707 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:24.980733 (+ 3026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:24.984617 (+ 3884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:24.984625 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:24.984643 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:24.985211 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:24.985688 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:24.985692 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:24.985696 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:24.986006 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:24.986657 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:24.986667 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:24.987070 (+ 403us) client_negotiation.cc:769] Sending connection context
0811 22:25:24.987415 (+ 345us) client_negotiation.cc:240] Negotiation successful
0811 22:25:24.987936 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":383,"threads_started":1}
I20260811 22:25:24.990382 1059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:24.966358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:53435 (local address 127.29.106.62:40297)
0811 22:25:24.966550 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:24.966556 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:24.967919 (+ 1363us) server_negotiation.cc:407] Connection header received
0811 22:25:24.970033 (+ 2114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:24.970040 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:24.970132 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:24.970239 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:24.972571 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:24.973387 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:24.975734 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:24.976298 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:24.976421 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:24.980960 (+ 4539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:24.980967 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:24.980973 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:24.981019 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:24.984468 (+ 3449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:24.985383 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:24.985389 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:24.985393 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:24.985546 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:24.986121 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:24.986127 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:24.986131 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:24.986527 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:24.986673 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:24.989839 (+ 3166us) server_negotiation.cc:299] Negotiation successful
0811 22:25:24.990049 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260811 22:25:25.007477 1574 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.3:42937
I20260811 22:25:25.007489 1847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.3:42937 every 8 connection(s)
I20260811 22:25:25.008920 1574 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
I20260811 22:25:25.010604 1059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:24.996308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:48003 (local address 127.29.106.62:40297)
0811 22:25:24.996446 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:24.996450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:24.997072 (+ 622us) server_negotiation.cc:407] Connection header received
0811 22:25:24.997223 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:24.997228 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:24.997284 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:24.997359 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:24.998311 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:24.999148 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:25.001370 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:25.001781 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:25.001888 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:25.004086 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:25.004093 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:25.004098 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:25.004131 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:25.007783 (+ 3652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:25.008399 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:25.008405 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:25.008408 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:25.008586 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:25.008986 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:25.008991 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:25.008993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:25.009332 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:25.009495 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:25.010047 (+ 552us) server_negotiation.cc:299] Negotiation successful
0811 22:25:25.010273 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260811 22:25:25.010974 1804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:24.996425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:40297 (local address 127.29.106.3:48003)
0811 22:25:24.996974 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:24.996989 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:24.997114 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:24.997412 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:24.997418 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:24.997862 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:24.998173 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:24.998183 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:24.999315 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:24.999321 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:25.001214 (+ 1893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:25.001232 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:25.001984 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:25.001991 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:25.002169 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:25.002744 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:25.002767 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:25.003892 (+ 1125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:25.007895 (+ 4003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:25.007898 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:25.007901 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:25.008283 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:25.008734 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:25.008738 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:25.008741 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:25.008866 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:25.009477 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:25.009482 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:25.009714 (+ 232us) client_negotiation.cc:769] Sending connection context
0811 22:25:25.009882 (+ 168us) client_negotiation.cc:240] Negotiation successful
0811 22:25:25.010573 (+ 691us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":507,"threads_started":1}
I20260811 22:25:25.011372 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 1574
I20260811 22:25:25.011399 1849 heartbeater.cc:347] Connected to a master server at 127.29.106.62:40297
I20260811 22:25:25.011540 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance
I20260811 22:25:25.011600 1849 heartbeater.cc:464] Registering TS with master...
I20260811 22:25:25.012001 30120 external_mini_cluster.cc:1428] Setting key c3448fa95e3bab6c4bbdacca58ce3c69
I20260811 22:25:25.012382 1849 heartbeater.cc:511] Master 127.29.106.62:40297 requested a full tablet report, sending...
I20260811 22:25:25.013909 876 ts_manager.cc:194] Registered new tserver with Master: eb362cc342004cdc9d75207b00cbee35 (127.29.106.3:42937)
I20260811 22:25:25.015025 876 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.3@KRBTEST.COM'} at 127.29.106.3:53435
I20260811 22:25:25.016556 30120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 22:25:25.021818 1277 heartbeater.cc:503] Master 127.29.106.62:40297 was elected leader, sending a full tablet report...
Aug 11 22:25:25 dist-test-slave-gjrz krb5kdc[778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 11 22:25:25 dist-test-slave-gjrz krb5kdc[778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:25:25.064575 1059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:25.051451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40140 (local address 127.29.106.62:40297)
0811 22:25:25.051667 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:25.051674 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:25.051779 (+ 105us) server_negotiation.cc:407] Connection header received
0811 22:25:25.051978 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:25.051984 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:25.052073 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:25.052162 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:25.053530 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:25.054262 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:25.055546 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:25.055958 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:25.056071 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:25.059743 (+ 3672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:25.059748 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:25.059751 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:25.059783 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:25.062072 (+ 2289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:25.062680 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:25.062684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:25.062686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:25.062809 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:25.063171 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:25.063174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:25.063176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:25.063494 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:25.063724 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:25.064015 (+ 291us) server_negotiation.cc:299] Negotiation successful
0811 22:25:25.064244 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260811 22:25:25.068449 876 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40140:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:25:25.075232 876 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:25:25.108968 1879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:25.086867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.3:42937 (local address 127.0.0.1:37306)
0811 22:25:25.100442 (+ 13575us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:25.100482 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:25.100667 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:25.101674 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:25.101682 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:25.101711 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:25:25.102151 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:25.102177 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:25.104259 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:25.104264 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:25.105827 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:25.105839 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:25.108070 (+ 2231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:25.108076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:25.108260 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:25.108312 (+ 52us) client_negotiation.cc:769] Sending connection context
0811 22:25:25.108454 (+ 142us) client_negotiation.cc:240] Negotiation successful
0811 22:25:25.108633 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13341,"thread_start_us":14080,"threads_started":1}
I20260811 22:25:25.108995 1875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:25.087151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37306 (local address 127.29.106.3:42937)
0811 22:25:25.099723 (+ 12572us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:25.099733 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:25.100601 (+ 868us) server_negotiation.cc:407] Connection header received
0811 22:25:25.100877 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:25.100887 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:25.101163 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:25.101365 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:25:25.102443 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:25.104119 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:25.106012 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:25.107826 (+ 1814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:25.108001 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:25.108238 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:25.108497 (+ 259us) server_negotiation.cc:299] Negotiation successful
0811 22:25:25.108671 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12360,"thread_start_us":12580,"threads_started":1}
I20260811 22:25:25.109167 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:25.087677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39668 (local address 127.29.106.2:37479)
0811 22:25:25.100033 (+ 12356us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:25.100041 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:25.101493 (+ 1452us) server_negotiation.cc:407] Connection header received
0811 22:25:25.101598 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:25.101606 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:25.101836 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:25.102101 (+ 265us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:25:25.102914 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:25.104270 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:25.106541 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:25.107911 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:25.108058 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:25.108254 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:25.108740 (+ 486us) server_negotiation.cc:299] Negotiation successful
0811 22:25:25.108899 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12184,"thread_start_us":12374,"threads_started":1}
I20260811 22:25:25.109839 1874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:25.087444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.2:37479 (local address 127.0.0.1:39668)
0811 22:25:25.101236 (+ 13792us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:25.101255 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:25.101360 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:25.102104 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:25.102110 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:25.102140 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:25:25.102753 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:25.102764 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:25.104397 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:25.104402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:25.106387 (+ 1985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:25.106399 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:25.108396 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:25.108402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:25.108552 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:25.108583 (+ 31us) client_negotiation.cc:769] Sending connection context
0811 22:25:25.109390 (+ 807us) client_negotiation.cc:240] Negotiation successful
0811 22:25:25.109516 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13670,"thread_start_us":12651,"threads_started":1}
I20260811 22:25:25.110318 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:25.101307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:33721 (local address 127.0.0.1:55836)
0811 22:25:25.102816 (+ 1509us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:25.102832 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:25.102945 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:25.103788 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:25.103793 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:25.103817 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:25:25.104164 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:25.104176 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:25.105956 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:25.105961 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:25.107937 (+ 1976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:25.107951 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:25.109675 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:25.109682 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:25.109858 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:25.109889 (+ 31us) client_negotiation.cc:769] Sending connection context
0811 22:25:25.109976 (+ 87us) client_negotiation.cc:240] Negotiation successful
0811 22:25:25.110093 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1380,"thread_start_us":633,"threads_started":1}
I20260811 22:25:25.110885 1882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:25.101493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55836 (local address 127.29.106.1:33721)
0811 22:25:25.101998 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:25.102006 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:25.103066 (+ 1060us) server_negotiation.cc:407] Connection header received
0811 22:25:25.103168 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:25.103175 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:25.103416 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:25.103598 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:25:25.104328 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:25.105812 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:25.108130 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:25.109473 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:25.109616 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:25.109807 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:25.110531 (+ 724us) server_negotiation.cc:299] Negotiation successful
0811 22:25:25.110666 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":222,"threads_started":1}
I20260811 22:25:25.112987 1707 tablet_service.cc:1467] Processing CreateTablet for tablet e26bf747b417404f925c0a8868323cc1 (DEFAULT_TABLE table=test-table [id=94fb2fa9bf3e44b5a82e1aaf3b39c627]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:25:25.113350 1427 tablet_service.cc:1467] Processing CreateTablet for tablet e26bf747b417404f925c0a8868323cc1 (DEFAULT_TABLE table=test-table [id=94fb2fa9bf3e44b5a82e1aaf3b39c627]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:25:25.114405 1707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e26bf747b417404f925c0a8868323cc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:25.114645 1427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e26bf747b417404f925c0a8868323cc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:25.114264 1136 tablet_service.cc:1467] Processing CreateTablet for tablet e26bf747b417404f925c0a8868323cc1 (DEFAULT_TABLE table=test-table [id=94fb2fa9bf3e44b5a82e1aaf3b39c627]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:25:25.115540 1136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e26bf747b417404f925c0a8868323cc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:25.126173 1884 tablet_bootstrap.cc:492] T e26bf747b417404f925c0a8868323cc1 P eb362cc342004cdc9d75207b00cbee35: Bootstrap starting.
I20260811 22:25:25.128321 1889 tablet_bootstrap.cc:492] T e26bf747b417404f925c0a8868323cc1 P a669f1e0074e4968bc0853b56071bdc0: Bootstrap starting.
I20260811 22:25:25.130020 1884 tablet_bootstrap.cc:654] T e26bf747b417404f925c0a8868323cc1 P eb362cc342004cdc9d75207b00cbee35: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:25.131353 1884 log.cc:824] T e26bf747b417404f925c0a8868323cc1 P eb362cc342004cdc9d75207b00cbee35: Log is configured to *not* fsync() on all Append() calls
I20260811 22:25:25.131620 1889 tablet_bootstrap.cc:654] T e26bf747b417404f925c0a8868323cc1 P a669f1e0074e4968bc0853b56071bdc0: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:25.132990 1889 log.cc:824] T e26bf747b417404f925c0a8868323cc1 P a669f1e0074e4968bc0853b56071bdc0: Log is configured to *not* fsync() on all Append() calls
I20260811 22:25:25.135614 1892 tablet_bootstrap.cc:492] T e26bf747b417404f925c0a8868323cc1 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: Bootstrap starting.
I20260811 22:25:25.136145 1889 tablet_bootstrap.cc:492] T e26bf747b417404f925c0a8868323cc1 P a669f1e0074e4968bc0853b56071bdc0: No bootstrap required, opened a new log
I20260811 22:25:25.136411 1889 ts_tablet_manager.cc:1397] T e26bf747b417404f925c0a8868323cc1 P a669f1e0074e4968bc0853b56071bdc0: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260811 22:25:25.138846 1892 tablet_bootstrap.cc:654] T e26bf747b417404f925c0a8868323cc1 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:25.140056 1892 log.cc:824] T e26bf747b417404f925c0a8868323cc1 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: Log is configured to *not* fsync() on all Append() calls
I20260811 22:25:25.141052 1884 tablet_bootstrap.cc:492] T e26bf747b417404f925c0a8868323cc1 P eb362cc342004cdc9d75207b00cbee35: No bootstrap required, opened a new log
I20260811 22:25:25.141359 1884 ts_tablet_manager.cc:1397] T e26bf747b417404f925c0a8868323cc1 P eb362cc342004cdc9d75207b00cbee35: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.004s
I20260811 22:25:25.146560 1889 raft_consensus.cc:348] T e26bf747b417404f925c0a8868323cc1 P a669f1e0074e4968bc0853b56071bdc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } } peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } }
I20260811 22:25:25.147069 1889 raft_consensus.cc:374] T e26bf747b417404f925c0a8868323cc1 P a669f1e0074e4968bc0853b56071bdc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:25:25.147215 1889 raft_consensus.cc:729] T e26bf747b417404f925c0a8868323cc1 P a669f1e0074e4968bc0853b56071bdc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a669f1e0074e4968bc0853b56071bdc0, State: Initialized, Role: FOLLOWER
I20260811 22:25:25.147521 1892 tablet_bootstrap.cc:492] T e26bf747b417404f925c0a8868323cc1 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: No bootstrap required, opened a new log
I20260811 22:25:25.147787 1892 ts_tablet_manager.cc:1397] T e26bf747b417404f925c0a8868323cc1 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260811 22:25:25.148008 1889 consensus_queue.cc:260] T e26bf747b417404f925c0a8868323cc1 P a669f1e0074e4968bc0853b56071bdc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } } peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } }
I20260811 22:25:25.149178 1567 heartbeater.cc:503] Master 127.29.106.62:40297 was elected leader, sending a full tablet report...
I20260811 22:25:25.149583 1884 raft_consensus.cc:348] T e26bf747b417404f925c0a8868323cc1 P eb362cc342004cdc9d75207b00cbee35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } } peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } }
I20260811 22:25:25.149824 1889 ts_tablet_manager.cc:1428] T e26bf747b417404f925c0a8868323cc1 P a669f1e0074e4968bc0853b56071bdc0: Time spent starting tablet: real 0.013s user 0.010s sys 0.003s
I20260811 22:25:25.149909 1884 raft_consensus.cc:374] T e26bf747b417404f925c0a8868323cc1 P eb362cc342004cdc9d75207b00cbee35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:25:25.150004 1884 raft_consensus.cc:729] T e26bf747b417404f925c0a8868323cc1 P eb362cc342004cdc9d75207b00cbee35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb362cc342004cdc9d75207b00cbee35, State: Initialized, Role: FOLLOWER
I20260811 22:25:25.150560 1884 consensus_queue.cc:260] T e26bf747b417404f925c0a8868323cc1 P eb362cc342004cdc9d75207b00cbee35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } } peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } }
I20260811 22:25:25.151594 1849 heartbeater.cc:503] Master 127.29.106.62:40297 was elected leader, sending a full tablet report...
I20260811 22:25:25.152151 1884 ts_tablet_manager.cc:1428] T e26bf747b417404f925c0a8868323cc1 P eb362cc342004cdc9d75207b00cbee35: Time spent starting tablet: real 0.011s user 0.005s sys 0.005s
I20260811 22:25:25.160226 1892 raft_consensus.cc:348] T e26bf747b417404f925c0a8868323cc1 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } } peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } }
I20260811 22:25:25.160646 1892 raft_consensus.cc:374] T e26bf747b417404f925c0a8868323cc1 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:25:25.160753 1892 raft_consensus.cc:729] T e26bf747b417404f925c0a8868323cc1 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bb5c6c3715e4f7ebbcca58e9cf0e7f5, State: Initialized, Role: FOLLOWER
I20260811 22:25:25.161257 1892 consensus_queue.cc:260] T e26bf747b417404f925c0a8868323cc1 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } } peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } }
I20260811 22:25:25.162555 1892 ts_tablet_manager.cc:1428] T e26bf747b417404f925c0a8868323cc1 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: Time spent starting tablet: real 0.015s user 0.010s sys 0.000s
W20260811 22:25:25.261142 1852 tablet.cc:2369] T e26bf747b417404f925c0a8868323cc1 P eb362cc342004cdc9d75207b00cbee35: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 22:25:25.265779 1278 tablet.cc:2369] T e26bf747b417404f925c0a8868323cc1 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:25:25.277966 1908 raft_consensus.cc:482] T e26bf747b417404f925c0a8868323cc1 P eb362cc342004cdc9d75207b00cbee35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:25:25.278244 1908 raft_consensus.cc:504] T e26bf747b417404f925c0a8868323cc1 P eb362cc342004cdc9d75207b00cbee35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } } peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } }
W20260811 22:25:25.278482 1569 tablet.cc:2369] T e26bf747b417404f925c0a8868323cc1 P a669f1e0074e4968bc0853b56071bdc0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:25:25.279911 1908 leader_election.cc:288] T e26bf747b417404f925c0a8868323cc1 P eb362cc342004cdc9d75207b00cbee35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 (127.29.106.1:33721), a669f1e0074e4968bc0853b56071bdc0 (127.29.106.2:37479)
I20260811 22:25:25.285300 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:25.280489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:40669 (local address 127.29.106.2:37479)
0811 22:25:25.280700 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:25.280706 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:25.280724 (+ 18us) server_negotiation.cc:407] Connection header received
0811 22:25:25.280788 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:25.280791 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:25.280848 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:25.280925 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:25:25.281394 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:25.282129 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:25.283727 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:25.284473 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:25.284618 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:25.284693 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:25.284931 (+ 238us) server_negotiation.cc:299] Negotiation successful
0811 22:25:25.285052 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260811 22:25:25.285346 1627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:25.280164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.2:37479 (local address 127.29.106.3:40669)
0811 22:25:25.280460 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:25.280483 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:25.280613 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:25.280970 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:25.280976 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:25.281004 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:25:25.281245 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:25.281255 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:25.282266 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:25.282276 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:25.283567 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:25.283580 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:25.284618 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:25.284624 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:25.284788 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:25.284812 (+ 24us) client_negotiation.cc:769] Sending connection context
0811 22:25:25.284901 (+ 89us) client_negotiation.cc:240] Negotiation successful
0811 22:25:25.285031 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"spinlock_wait_cycles":44416}
I20260811 22:25:25.285828 1882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:25.280407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:39425 (local address 127.29.106.1:33721)
0811 22:25:25.280612 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:25.280619 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:25.281024 (+ 405us) server_negotiation.cc:407] Connection header received
0811 22:25:25.281214 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:25.281218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:25.281275 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:25.281350 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:25:25.281862 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:25.282565 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:25.284201 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:25.284933 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:25.285071 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:25.285140 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:25.285512 (+ 372us) server_negotiation.cc:299] Negotiation successful
0811 22:25:25.285632 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260811 22:25:25.286454 1911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:25.280257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:33721 (local address 127.29.106.3:39425)
0811 22:25:25.280912 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:25.280927 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:25.281063 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:25.281379 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:25.281383 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:25.281406 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:25:25.281703 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:25.281711 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:25.282696 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:25.282699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:25.284054 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:25.284066 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:25.285188 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:25.285195 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:25.285334 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:25.285360 (+ 26us) client_negotiation.cc:769] Sending connection context
0811 22:25:25.285995 (+ 635us) client_negotiation.cc:240] Negotiation successful
0811 22:25:25.286131 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":764,"threads_started":1}
I20260811 22:25:25.286401 1468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e26bf747b417404f925c0a8868323cc1" candidate_uuid: "eb362cc342004cdc9d75207b00cbee35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a669f1e0074e4968bc0853b56071bdc0" is_pre_election: true
I20260811 22:25:25.286928 1468 raft_consensus.cc:2454] T e26bf747b417404f925c0a8868323cc1 P a669f1e0074e4968bc0853b56071bdc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb362cc342004cdc9d75207b00cbee35 in term 0.
I20260811 22:25:25.287465 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e26bf747b417404f925c0a8868323cc1" candidate_uuid: "eb362cc342004cdc9d75207b00cbee35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" is_pre_election: true
I20260811 22:25:25.287621 1610 leader_election.cc:302] T e26bf747b417404f925c0a8868323cc1 P eb362cc342004cdc9d75207b00cbee35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a669f1e0074e4968bc0853b56071bdc0, eb362cc342004cdc9d75207b00cbee35; no voters:
I20260811 22:25:25.287883 1189 raft_consensus.cc:2454] T e26bf747b417404f925c0a8868323cc1 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb362cc342004cdc9d75207b00cbee35 in term 0.
I20260811 22:25:25.288010 1908 raft_consensus.cc:2781] T e26bf747b417404f925c0a8868323cc1 P eb362cc342004cdc9d75207b00cbee35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:25:25.288162 1908 raft_consensus.cc:482] T e26bf747b417404f925c0a8868323cc1 P eb362cc342004cdc9d75207b00cbee35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:25:25.288268 1908 raft_consensus.cc:3037] T e26bf747b417404f925c0a8868323cc1 P eb362cc342004cdc9d75207b00cbee35 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:25.290397 1908 raft_consensus.cc:504] T e26bf747b417404f925c0a8868323cc1 P eb362cc342004cdc9d75207b00cbee35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } } peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } }
I20260811 22:25:25.290969 1908 leader_election.cc:288] T e26bf747b417404f925c0a8868323cc1 P eb362cc342004cdc9d75207b00cbee35 [CANDIDATE]: Term 1 election: Requested vote from peers 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 (127.29.106.1:33721), a669f1e0074e4968bc0853b56071bdc0 (127.29.106.2:37479)
I20260811 22:25:25.291365 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e26bf747b417404f925c0a8868323cc1" candidate_uuid: "eb362cc342004cdc9d75207b00cbee35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5"
I20260811 22:25:25.291467 1468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e26bf747b417404f925c0a8868323cc1" candidate_uuid: "eb362cc342004cdc9d75207b00cbee35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a669f1e0074e4968bc0853b56071bdc0"
I20260811 22:25:25.291577 1189 raft_consensus.cc:3037] T e26bf747b417404f925c0a8868323cc1 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:25.291653 1468 raft_consensus.cc:3037] T e26bf747b417404f925c0a8868323cc1 P a669f1e0074e4968bc0853b56071bdc0 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:25.294691 1189 raft_consensus.cc:2454] T e26bf747b417404f925c0a8868323cc1 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb362cc342004cdc9d75207b00cbee35 in term 1.
I20260811 22:25:25.294696 1468 raft_consensus.cc:2454] T e26bf747b417404f925c0a8868323cc1 P a669f1e0074e4968bc0853b56071bdc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb362cc342004cdc9d75207b00cbee35 in term 1.
I20260811 22:25:25.295058 1610 leader_election.cc:302] T e26bf747b417404f925c0a8868323cc1 P eb362cc342004cdc9d75207b00cbee35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a669f1e0074e4968bc0853b56071bdc0, eb362cc342004cdc9d75207b00cbee35; no voters:
I20260811 22:25:25.295351 1908 raft_consensus.cc:2781] T e26bf747b417404f925c0a8868323cc1 P eb362cc342004cdc9d75207b00cbee35 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:25:25.295724 1908 raft_consensus.cc:686] T e26bf747b417404f925c0a8868323cc1 P eb362cc342004cdc9d75207b00cbee35 [term 1 LEADER]: Becoming Leader. State: Replica: eb362cc342004cdc9d75207b00cbee35, State: Running, Role: LEADER
I20260811 22:25:25.296160 1908 consensus_queue.cc:237] T e26bf747b417404f925c0a8868323cc1 P eb362cc342004cdc9d75207b00cbee35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } } peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } }
I20260811 22:25:25.300700 877 catalog_manager.cc:5495] T e26bf747b417404f925c0a8868323cc1 P eb362cc342004cdc9d75207b00cbee35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb362cc342004cdc9d75207b00cbee35 (127.29.106.3). New cstate: current_term: 1 leader_uuid: "eb362cc342004cdc9d75207b00cbee35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } health_report { overall_health: UNKNOWN } } }
Aug 11 22:25:25 dist-test-slave-gjrz krb5kdc[778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487123, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:25:25.378343 1059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:25.365433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40146 (local address 127.29.106.62:40297)
0811 22:25:25.365594 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:25.365599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:25.366164 (+ 565us) server_negotiation.cc:407] Connection header received
0811 22:25:25.366320 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:25.366326 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:25.366395 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:25.366473 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:25.367682 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:25.368401 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:25.369640 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:25.370066 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:25.370168 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:25.373228 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:25.373233 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:25.373236 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:25.373265 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:25.375810 (+ 2545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:25.376443 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:25.376448 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:25.376451 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:25.376587 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:25.376925 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:25.376929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:25.376931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:25.377202 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:25.377330 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:25.377764 (+ 434us) server_negotiation.cc:299] Negotiation successful
0811 22:25:25.378049 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":86,"server-negotiator.queue_time_us":49}
I20260811 22:25:25.378376 1932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:25.365520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:40297 (local address 127.0.0.1:40146)
0811 22:25:25.366065 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:25.366080 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:25.366215 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:25.366525 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:25.366529 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:25.367211 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:25.367537 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:25.367550 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:25.368531 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:25.368535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:25.369509 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:25.369523 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:25.370233 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:25.370238 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:25.370388 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:25.370942 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:25.370958 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:25.373075 (+ 2117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:25.375933 (+ 2858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:25.375946 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:25.375954 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:25.376338 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:25.376699 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:25.376703 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:25.376707 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:25.376829 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:25.377330 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:25.377338 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:25.377488 (+ 150us) client_negotiation.cc:769] Sending connection context
0811 22:25:25.377636 (+ 148us) client_negotiation.cc:240] Negotiation successful
0811 22:25:25.377909 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":256,"threads_started":1}
I20260811 22:25:25.383035 877 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:40146:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260811 22:25:25.384069 877 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:25:25.390309 1427 tablet_service.cc:1467] Processing CreateTablet for tablet 087d10a784e04413bd4049678cbd815f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5b282dfd4a654345a8e401c42a526a5f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 22:25:25.390223 1136 tablet_service.cc:1467] Processing CreateTablet for tablet 087d10a784e04413bd4049678cbd815f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5b282dfd4a654345a8e401c42a526a5f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 22:25:25.391003 1136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 087d10a784e04413bd4049678cbd815f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:25.391017 1427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 087d10a784e04413bd4049678cbd815f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:25.391745 1707 tablet_service.cc:1467] Processing CreateTablet for tablet 087d10a784e04413bd4049678cbd815f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5b282dfd4a654345a8e401c42a526a5f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 22:25:25.392410 1707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 087d10a784e04413bd4049678cbd815f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:25.396281 1889 tablet_bootstrap.cc:492] T 087d10a784e04413bd4049678cbd815f P a669f1e0074e4968bc0853b56071bdc0: Bootstrap starting.
I20260811 22:25:25.397459 1884 tablet_bootstrap.cc:492] T 087d10a784e04413bd4049678cbd815f P eb362cc342004cdc9d75207b00cbee35: Bootstrap starting.
I20260811 22:25:25.397955 1889 tablet_bootstrap.cc:654] T 087d10a784e04413bd4049678cbd815f P a669f1e0074e4968bc0853b56071bdc0: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:25.399029 1884 tablet_bootstrap.cc:654] T 087d10a784e04413bd4049678cbd815f P eb362cc342004cdc9d75207b00cbee35: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:25.400225 1889 tablet_bootstrap.cc:492] T 087d10a784e04413bd4049678cbd815f P a669f1e0074e4968bc0853b56071bdc0: No bootstrap required, opened a new log
I20260811 22:25:25.400527 1889 ts_tablet_manager.cc:1397] T 087d10a784e04413bd4049678cbd815f P a669f1e0074e4968bc0853b56071bdc0: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260811 22:25:25.401121 1889 raft_consensus.cc:348] T 087d10a784e04413bd4049678cbd815f P a669f1e0074e4968bc0853b56071bdc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } } peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } }
I20260811 22:25:25.401304 1889 raft_consensus.cc:374] T 087d10a784e04413bd4049678cbd815f P a669f1e0074e4968bc0853b56071bdc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:25:25.401389 1889 raft_consensus.cc:729] T 087d10a784e04413bd4049678cbd815f P a669f1e0074e4968bc0853b56071bdc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a669f1e0074e4968bc0853b56071bdc0, State: Initialized, Role: FOLLOWER
I20260811 22:25:25.401613 1889 consensus_queue.cc:260] T 087d10a784e04413bd4049678cbd815f P a669f1e0074e4968bc0853b56071bdc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } } peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } }
I20260811 22:25:25.402016 1901 tablet_replica.cc:439] T 087d10a784e04413bd4049678cbd815f P a669f1e0074e4968bc0853b56071bdc0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } } peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } } }
I20260811 22:25:25.402396 1901 tablet_replica.cc:442] T 087d10a784e04413bd4049678cbd815f P a669f1e0074e4968bc0853b56071bdc0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 22:25:25.402848 1889 ts_tablet_manager.cc:1428] T 087d10a784e04413bd4049678cbd815f P a669f1e0074e4968bc0853b56071bdc0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260811 22:25:25.403434 1892 tablet_bootstrap.cc:492] T 087d10a784e04413bd4049678cbd815f P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: Bootstrap starting.
I20260811 22:25:25.404844 1892 tablet_bootstrap.cc:654] T 087d10a784e04413bd4049678cbd815f P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:25.410466 1884 tablet_bootstrap.cc:492] T 087d10a784e04413bd4049678cbd815f P eb362cc342004cdc9d75207b00cbee35: No bootstrap required, opened a new log
I20260811 22:25:25.411167 1884 ts_tablet_manager.cc:1397] T 087d10a784e04413bd4049678cbd815f P eb362cc342004cdc9d75207b00cbee35: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.005s
I20260811 22:25:25.411963 1884 raft_consensus.cc:348] T 087d10a784e04413bd4049678cbd815f P eb362cc342004cdc9d75207b00cbee35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } } peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } }
I20260811 22:25:25.412153 1884 raft_consensus.cc:374] T 087d10a784e04413bd4049678cbd815f P eb362cc342004cdc9d75207b00cbee35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:25:25.412220 1892 tablet_bootstrap.cc:492] T 087d10a784e04413bd4049678cbd815f P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: No bootstrap required, opened a new log
I20260811 22:25:25.412240 1884 raft_consensus.cc:729] T 087d10a784e04413bd4049678cbd815f P eb362cc342004cdc9d75207b00cbee35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb362cc342004cdc9d75207b00cbee35, State: Initialized, Role: FOLLOWER
I20260811 22:25:25.412359 1892 ts_tablet_manager.cc:1397] T 087d10a784e04413bd4049678cbd815f P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260811 22:25:25.412463 1884 consensus_queue.cc:260] T 087d10a784e04413bd4049678cbd815f P eb362cc342004cdc9d75207b00cbee35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } } peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } }
I20260811 22:25:25.412856 1908 tablet_replica.cc:439] T 087d10a784e04413bd4049678cbd815f P eb362cc342004cdc9d75207b00cbee35: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } } peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } } }
I20260811 22:25:25.413053 1884 ts_tablet_manager.cc:1428] T 087d10a784e04413bd4049678cbd815f P eb362cc342004cdc9d75207b00cbee35: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260811 22:25:25.413161 1908 tablet_replica.cc:442] T 087d10a784e04413bd4049678cbd815f P eb362cc342004cdc9d75207b00cbee35: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 22:25:25.413056 1892 raft_consensus.cc:348] T 087d10a784e04413bd4049678cbd815f P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } } peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } }
I20260811 22:25:25.413295 1892 raft_consensus.cc:374] T 087d10a784e04413bd4049678cbd815f P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:25:25.413401 1892 raft_consensus.cc:729] T 087d10a784e04413bd4049678cbd815f P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bb5c6c3715e4f7ebbcca58e9cf0e7f5, State: Initialized, Role: FOLLOWER
I20260811 22:25:25.413650 1892 consensus_queue.cc:260] T 087d10a784e04413bd4049678cbd815f P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } } peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } }
I20260811 22:25:25.414196 1892 ts_tablet_manager.cc:1428] T 087d10a784e04413bd4049678cbd815f P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260811 22:25:25.414650 1909 tablet_replica.cc:439] T 087d10a784e04413bd4049678cbd815f P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } } peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } } }
I20260811 22:25:25.415002 1909 tablet_replica.cc:442] T 087d10a784e04413bd4049678cbd815f P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 22:25:25.507942 1908 raft_consensus.cc:482] T 087d10a784e04413bd4049678cbd815f P eb362cc342004cdc9d75207b00cbee35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:25:25.508172 1908 raft_consensus.cc:504] T 087d10a784e04413bd4049678cbd815f P eb362cc342004cdc9d75207b00cbee35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } } peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } }
I20260811 22:25:25.508809 1908 leader_election.cc:288] T 087d10a784e04413bd4049678cbd815f P eb362cc342004cdc9d75207b00cbee35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 (127.29.106.1:33721), a669f1e0074e4968bc0853b56071bdc0 (127.29.106.2:37479)
I20260811 22:25:25.509328 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "087d10a784e04413bd4049678cbd815f" candidate_uuid: "eb362cc342004cdc9d75207b00cbee35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" is_pre_election: true
I20260811 22:25:25.509336 1468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "087d10a784e04413bd4049678cbd815f" candidate_uuid: "eb362cc342004cdc9d75207b00cbee35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a669f1e0074e4968bc0853b56071bdc0" is_pre_election: true
I20260811 22:25:25.509568 1189 raft_consensus.cc:2454] T 087d10a784e04413bd4049678cbd815f P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb362cc342004cdc9d75207b00cbee35 in term 0.
I20260811 22:25:25.509577 1468 raft_consensus.cc:2454] T 087d10a784e04413bd4049678cbd815f P a669f1e0074e4968bc0853b56071bdc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb362cc342004cdc9d75207b00cbee35 in term 0.
I20260811 22:25:25.509960 1617 leader_election.cc:302] T 087d10a784e04413bd4049678cbd815f P eb362cc342004cdc9d75207b00cbee35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bb5c6c3715e4f7ebbcca58e9cf0e7f5, eb362cc342004cdc9d75207b00cbee35; no voters:
I20260811 22:25:25.510253 1908 raft_consensus.cc:2781] T 087d10a784e04413bd4049678cbd815f P eb362cc342004cdc9d75207b00cbee35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:25:25.510354 1908 raft_consensus.cc:482] T 087d10a784e04413bd4049678cbd815f P eb362cc342004cdc9d75207b00cbee35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:25:25.510464 1908 raft_consensus.cc:3037] T 087d10a784e04413bd4049678cbd815f P eb362cc342004cdc9d75207b00cbee35 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:25.512660 1908 raft_consensus.cc:504] T 087d10a784e04413bd4049678cbd815f P eb362cc342004cdc9d75207b00cbee35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } } peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } }
I20260811 22:25:25.513247 1908 leader_election.cc:288] T 087d10a784e04413bd4049678cbd815f P eb362cc342004cdc9d75207b00cbee35 [CANDIDATE]: Term 1 election: Requested vote from peers 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 (127.29.106.1:33721), a669f1e0074e4968bc0853b56071bdc0 (127.29.106.2:37479)
I20260811 22:25:25.513562 1189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "087d10a784e04413bd4049678cbd815f" candidate_uuid: "eb362cc342004cdc9d75207b00cbee35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5"
I20260811 22:25:25.513662 1468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "087d10a784e04413bd4049678cbd815f" candidate_uuid: "eb362cc342004cdc9d75207b00cbee35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a669f1e0074e4968bc0853b56071bdc0"
I20260811 22:25:25.513746 1189 raft_consensus.cc:3037] T 087d10a784e04413bd4049678cbd815f P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:25.513877 1468 raft_consensus.cc:3037] T 087d10a784e04413bd4049678cbd815f P a669f1e0074e4968bc0853b56071bdc0 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:25.515761 1189 raft_consensus.cc:2454] T 087d10a784e04413bd4049678cbd815f P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb362cc342004cdc9d75207b00cbee35 in term 1.
I20260811 22:25:25.516129 1617 leader_election.cc:302] T 087d10a784e04413bd4049678cbd815f P eb362cc342004cdc9d75207b00cbee35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bb5c6c3715e4f7ebbcca58e9cf0e7f5, eb362cc342004cdc9d75207b00cbee35; no voters:
I20260811 22:25:25.516191 1468 raft_consensus.cc:2454] T 087d10a784e04413bd4049678cbd815f P a669f1e0074e4968bc0853b56071bdc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb362cc342004cdc9d75207b00cbee35 in term 1.
I20260811 22:25:25.516381 1908 raft_consensus.cc:2781] T 087d10a784e04413bd4049678cbd815f P eb362cc342004cdc9d75207b00cbee35 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:25:25.516610 1908 raft_consensus.cc:686] T 087d10a784e04413bd4049678cbd815f P eb362cc342004cdc9d75207b00cbee35 [term 1 LEADER]: Becoming Leader. State: Replica: eb362cc342004cdc9d75207b00cbee35, State: Running, Role: LEADER
I20260811 22:25:25.516855 1908 consensus_queue.cc:237] T 087d10a784e04413bd4049678cbd815f P eb362cc342004cdc9d75207b00cbee35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } } peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } }
I20260811 22:25:25.518380 1915 tablet_replica.cc:439] T 087d10a784e04413bd4049678cbd815f P eb362cc342004cdc9d75207b00cbee35: TxnStatusTablet state changed. Reason: New leader eb362cc342004cdc9d75207b00cbee35. Latest consensus state: current_term: 1 leader_uuid: "eb362cc342004cdc9d75207b00cbee35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } } peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } } }
I20260811 22:25:25.518657 1915 tablet_replica.cc:442] T 087d10a784e04413bd4049678cbd815f P eb362cc342004cdc9d75207b00cbee35: This TxnStatusTablet replica's current role is: LEADER
I20260811 22:25:25.519243 1939 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260811 22:25:25.519507 1939 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260811 22:25:25.519783 877 catalog_manager.cc:5495] T 087d10a784e04413bd4049678cbd815f P eb362cc342004cdc9d75207b00cbee35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb362cc342004cdc9d75207b00cbee35 (127.29.106.3). New cstate: current_term: 1 leader_uuid: "eb362cc342004cdc9d75207b00cbee35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } health_report { overall_health: HEALTHY } } }
I20260811 22:25:25.563231 1932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:25.557044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.3:42937 (local address 127.0.0.1:37312)
0811 22:25:25.557584 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:25.557602 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:25.557712 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:25.558165 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:25.558168 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:25.558178 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 22:25:25.558337 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:25.558344 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:25.559734 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:25.559738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:25.560728 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:25.560740 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:25.561690 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:25.561695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:25.561853 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:25.561921 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 22:25:25.562825 (+ 904us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 22:25:25.562846 (+ 21us) client_negotiation.cc:769] Sending connection context
0811 22:25:25.562901 (+ 55us) client_negotiation.cc:240] Negotiation successful
0811 22:25:25.563019 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351}
I20260811 22:25:25.563304 1875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:25.557530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37312 (local address 127.29.106.3:42937)
0811 22:25:25.557713 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:25.557719 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:25.557741 (+ 22us) server_negotiation.cc:407] Connection header received
0811 22:25:25.557884 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:25.557890 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:25.557990 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:25.558144 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:25:25.558457 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:25.559610 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:25.560872 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:25.561548 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:25.561679 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:25.562116 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:25:25.562726 (+ 610us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:25:25.562869 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:25.562923 (+ 54us) server_negotiation.cc:299] Negotiation successful
0811 22:25:25.563008 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260811 22:25:25.571099 1189 raft_consensus.cc:1261] T 087d10a784e04413bd4049678cbd815f P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [term 1 FOLLOWER]: Refusing update from remote peer eb362cc342004cdc9d75207b00cbee35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:25:25.571178 1468 raft_consensus.cc:1261] T 087d10a784e04413bd4049678cbd815f P a669f1e0074e4968bc0853b56071bdc0 [term 1 FOLLOWER]: Refusing update from remote peer eb362cc342004cdc9d75207b00cbee35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:25:25.572064 1915 consensus_queue.cc:1035] T 087d10a784e04413bd4049678cbd815f P eb362cc342004cdc9d75207b00cbee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:25:25.572434 1908 consensus_queue.cc:1035] T 087d10a784e04413bd4049678cbd815f P eb362cc342004cdc9d75207b00cbee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:25:25.581270 1901 tablet_replica.cc:439] T 087d10a784e04413bd4049678cbd815f P a669f1e0074e4968bc0853b56071bdc0: TxnStatusTablet state changed. Reason: New leader eb362cc342004cdc9d75207b00cbee35. Latest consensus state: current_term: 1 leader_uuid: "eb362cc342004cdc9d75207b00cbee35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } } peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } } }
I20260811 22:25:25.581555 1901 tablet_replica.cc:442] T 087d10a784e04413bd4049678cbd815f P a669f1e0074e4968bc0853b56071bdc0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 22:25:25.583506 1944 mvcc.cc:205] Tried to move back new op lower bound from 7317451266330546176 to 7317451266119372800. Current Snapshot: MvccSnapshot[applied={T|T < 7317451266330546176}]
I20260811 22:25:25.584666 1948 mvcc.cc:205] Tried to move back new op lower bound from 7317451266330546176 to 7317451266119372800. Current Snapshot: MvccSnapshot[applied={T|T < 7317451266330546176}]
I20260811 22:25:25.585259 1909 tablet_replica.cc:439] T 087d10a784e04413bd4049678cbd815f P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: TxnStatusTablet state changed. Reason: New leader eb362cc342004cdc9d75207b00cbee35. Latest consensus state: current_term: 1 leader_uuid: "eb362cc342004cdc9d75207b00cbee35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } } peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } } }
I20260811 22:25:25.585553 1909 tablet_replica.cc:442] T 087d10a784e04413bd4049678cbd815f P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 22:25:25.587574 1901 tablet_replica.cc:439] T 087d10a784e04413bd4049678cbd815f P a669f1e0074e4968bc0853b56071bdc0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eb362cc342004cdc9d75207b00cbee35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } } peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } } }
I20260811 22:25:25.587930 1901 tablet_replica.cc:442] T 087d10a784e04413bd4049678cbd815f P a669f1e0074e4968bc0853b56071bdc0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 22:25:25.591177 1908 tablet_replica.cc:439] T 087d10a784e04413bd4049678cbd815f P eb362cc342004cdc9d75207b00cbee35: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eb362cc342004cdc9d75207b00cbee35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } } peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } } }
I20260811 22:25:25.591732 1908 tablet_replica.cc:442] T 087d10a784e04413bd4049678cbd815f P eb362cc342004cdc9d75207b00cbee35: This TxnStatusTablet replica's current role is: LEADER
I20260811 22:25:25.591753 1909 tablet_replica.cc:439] T 087d10a784e04413bd4049678cbd815f P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eb362cc342004cdc9d75207b00cbee35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } } peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } } }
I20260811 22:25:25.591984 1909 tablet_replica.cc:442] T 087d10a784e04413bd4049678cbd815f P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 22:25:25.599081 1915 tablet_replica.cc:439] T 087d10a784e04413bd4049678cbd815f P eb362cc342004cdc9d75207b00cbee35: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eb362cc342004cdc9d75207b00cbee35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } } peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } } }
I20260811 22:25:25.599520 1915 tablet_replica.cc:442] T 087d10a784e04413bd4049678cbd815f P eb362cc342004cdc9d75207b00cbee35: This TxnStatusTablet replica's current role is: LEADER
I20260811 22:25:25.612823 1875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:25.603376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37320 (local address 127.29.106.3:42937)
0811 22:25:25.603561 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:25.603568 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:25.605073 (+ 1505us) server_negotiation.cc:407] Connection header received
0811 22:25:25.605250 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:25.605255 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:25.605331 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:25.605442 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:25:25.606221 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:25.607981 (+ 1760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:25.610283 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:25.611334 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:25.611503 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:25.611855 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:25:25.612025 (+ 170us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:25:25.612115 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:25.612246 (+ 131us) server_negotiation.cc:299] Negotiation successful
0811 22:25:25.612331 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260811 22:25:25.631978 1875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:25.626339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:42665 (local address 127.29.106.3:42937)
0811 22:25:25.626509 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:25.626515 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:25.627207 (+ 692us) server_negotiation.cc:407] Connection header received
0811 22:25:25.627411 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:25.627414 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:25.627478 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:25.627547 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:25:25.628056 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:25.628749 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:25.630186 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:25.630641 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:25.630748 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:25.631202 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:25:25.631350 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:25:25.631446 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:25.631553 (+ 107us) server_negotiation.cc:299] Negotiation successful
0811 22:25:25.631617 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260811 22:25:25.632464 1963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:25.626396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.3:42937 (local address 127.29.106.3:42665)
0811 22:25:25.627118 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:25.627134 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:25.627243 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:25.627578 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:25.627582 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:25.627592 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 22:25:25.627902 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:25.627912 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:25.628878 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:25.628882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:25.630059 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:25.630071 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:25.630828 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:25.630833 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:25.630962 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:25.631012 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 22:25:25.631445 (+ 433us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 22:25:25.631455 (+ 10us) client_negotiation.cc:769] Sending connection context
0811 22:25:25.631532 (+ 77us) client_negotiation.cc:240] Negotiation successful
0811 22:25:25.631656 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":401,"threads_started":1}
I20260811 22:25:25.641453 1180 raft_consensus.cc:1261] T e26bf747b417404f925c0a8868323cc1 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [term 1 FOLLOWER]: Refusing update from remote peer eb362cc342004cdc9d75207b00cbee35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:25:25.641436 1468 raft_consensus.cc:1261] T e26bf747b417404f925c0a8868323cc1 P a669f1e0074e4968bc0853b56071bdc0 [term 1 FOLLOWER]: Refusing update from remote peer eb362cc342004cdc9d75207b00cbee35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:25:25.642210 1915 consensus_queue.cc:1035] T e26bf747b417404f925c0a8868323cc1 P eb362cc342004cdc9d75207b00cbee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:25:25.642499 1908 consensus_queue.cc:1035] T e26bf747b417404f925c0a8868323cc1 P eb362cc342004cdc9d75207b00cbee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:25:25.700301 877 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:25:25.700615 877 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:25:25.703971 877 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc4020c4fbb7446aadc65b89aeb691ba: Sending DeleteTablet for 3 replicas of tablet e26bf747b417404f925c0a8868323cc1
I20260811 22:25:25.704965 1707 tablet_service.cc:1514] Processing DeleteTablet for tablet e26bf747b417404f925c0a8868323cc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:25:25 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:37306
I20260811 22:25:25.705087 1136 tablet_service.cc:1514] Processing DeleteTablet for tablet e26bf747b417404f925c0a8868323cc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:25:25 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:55836
I20260811 22:25:25.705531 1427 tablet_service.cc:1514] Processing DeleteTablet for tablet e26bf747b417404f925c0a8868323cc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:25:25 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:39668
I20260811 22:25:25.706867 877 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40140:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:25:25.707638 877 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:25:25.715224 1132 tablet_service.cc:1467] Processing CreateTablet for tablet 3e58ce0ecc3e4295994c1d7dba61e156 (DEFAULT_TABLE table=test-table [id=62baec03d271427995cbd4129f33a1f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:25:25.716184 1132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e58ce0ecc3e4295994c1d7dba61e156. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:25.718019 1970 tablet_replica.cc:330] T e26bf747b417404f925c0a8868323cc1 P eb362cc342004cdc9d75207b00cbee35: stopping tablet replica
I20260811 22:25:25.718554 1969 tablet_replica.cc:330] T e26bf747b417404f925c0a8868323cc1 P a669f1e0074e4968bc0853b56071bdc0: stopping tablet replica
I20260811 22:25:25.719135 1969 raft_consensus.cc:2229] T e26bf747b417404f925c0a8868323cc1 P a669f1e0074e4968bc0853b56071bdc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:25:25.719693 1970 raft_consensus.cc:2229] T e26bf747b417404f925c0a8868323cc1 P eb362cc342004cdc9d75207b00cbee35 [term 1 LEADER]: Raft consensus shutting down.
I20260811 22:25:25.720160 1969 raft_consensus.cc:2258] T e26bf747b417404f925c0a8868323cc1 P a669f1e0074e4968bc0853b56071bdc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:25:25.719987 1706 tablet_service.cc:1467] Processing CreateTablet for tablet 3e58ce0ecc3e4295994c1d7dba61e156 (DEFAULT_TABLE table=test-table [id=62baec03d271427995cbd4129f33a1f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:25:25.720655 1967 tablet_replica.cc:330] T e26bf747b417404f925c0a8868323cc1 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: stopping tablet replica
I20260811 22:25:25.720899 1706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e58ce0ecc3e4295994c1d7dba61e156. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:25.721040 1967 raft_consensus.cc:2229] T e26bf747b417404f925c0a8868323cc1 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:25:25.721230 1970 raft_consensus.cc:2258] T e26bf747b417404f925c0a8868323cc1 P eb362cc342004cdc9d75207b00cbee35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:25:25.721390 1967 raft_consensus.cc:2258] T e26bf747b417404f925c0a8868323cc1 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:25:25.723177 1967 ts_tablet_manager.cc:1905] T e26bf747b417404f925c0a8868323cc1 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:25:25.723177 1969 ts_tablet_manager.cc:1905] T e26bf747b417404f925c0a8868323cc1 P a669f1e0074e4968bc0853b56071bdc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:25:25.724895 1970 ts_tablet_manager.cc:1905] T e26bf747b417404f925c0a8868323cc1 P eb362cc342004cdc9d75207b00cbee35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:25:25.730460 1967 ts_tablet_manager.cc:1918] T e26bf747b417404f925c0a8868323cc1 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 22:25:25.730692 1967 log.cc:1196] T e26bf747b417404f925c0a8868323cc1 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-0/wal/wals/e26bf747b417404f925c0a8868323cc1
I20260811 22:25:25.731189 1892 tablet_bootstrap.cc:492] T 3e58ce0ecc3e4295994c1d7dba61e156 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: Bootstrap starting.
I20260811 22:25:25.731328 1967 ts_tablet_manager.cc:1939] T e26bf747b417404f925c0a8868323cc1 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: Deleting consensus metadata
I20260811 22:25:25.731827 1427 tablet_service.cc:1467] Processing CreateTablet for tablet 3e58ce0ecc3e4295994c1d7dba61e156 (DEFAULT_TABLE table=test-table [id=62baec03d271427995cbd4129f33a1f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:25:25.732209 1427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e58ce0ecc3e4295994c1d7dba61e156. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:25.732885 1892 tablet_bootstrap.cc:654] T 3e58ce0ecc3e4295994c1d7dba61e156 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:25.733464 850 catalog_manager.cc:4841] TS 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 (127.29.106.1:33721): tablet e26bf747b417404f925c0a8868323cc1 (table test-table [id=94fb2fa9bf3e44b5a82e1aaf3b39c627]) successfully deleted
I20260811 22:25:25.737002 1884 tablet_bootstrap.cc:492] T 3e58ce0ecc3e4295994c1d7dba61e156 P eb362cc342004cdc9d75207b00cbee35: Bootstrap starting.
I20260811 22:25:25.738617 1884 tablet_bootstrap.cc:654] T 3e58ce0ecc3e4295994c1d7dba61e156 P eb362cc342004cdc9d75207b00cbee35: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:25.742102 1969 ts_tablet_manager.cc:1918] T e26bf747b417404f925c0a8868323cc1 P a669f1e0074e4968bc0853b56071bdc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 22:25:25.742259 1969 log.cc:1196] T e26bf747b417404f925c0a8868323cc1 P a669f1e0074e4968bc0853b56071bdc0: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-1/wal/wals/e26bf747b417404f925c0a8868323cc1
I20260811 22:25:25.748818 1884 tablet_bootstrap.cc:492] T 3e58ce0ecc3e4295994c1d7dba61e156 P eb362cc342004cdc9d75207b00cbee35: No bootstrap required, opened a new log
I20260811 22:25:25.748982 1884 ts_tablet_manager.cc:1397] T 3e58ce0ecc3e4295994c1d7dba61e156 P eb362cc342004cdc9d75207b00cbee35: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260811 22:25:25.749097 1970 ts_tablet_manager.cc:1918] T e26bf747b417404f925c0a8868323cc1 P eb362cc342004cdc9d75207b00cbee35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 22:25:25.749286 1970 log.cc:1196] T e26bf747b417404f925c0a8868323cc1 P eb362cc342004cdc9d75207b00cbee35: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-2/wal/wals/e26bf747b417404f925c0a8868323cc1
I20260811 22:25:25.749888 1970 ts_tablet_manager.cc:1939] T e26bf747b417404f925c0a8868323cc1 P eb362cc342004cdc9d75207b00cbee35: Deleting consensus metadata
I20260811 22:25:25.749794 1884 raft_consensus.cc:348] T 3e58ce0ecc3e4295994c1d7dba61e156 P eb362cc342004cdc9d75207b00cbee35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } } peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } }
I20260811 22:25:25.750054 1884 raft_consensus.cc:374] T 3e58ce0ecc3e4295994c1d7dba61e156 P eb362cc342004cdc9d75207b00cbee35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:25:25.750141 1884 raft_consensus.cc:729] T 3e58ce0ecc3e4295994c1d7dba61e156 P eb362cc342004cdc9d75207b00cbee35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb362cc342004cdc9d75207b00cbee35, State: Initialized, Role: FOLLOWER
I20260811 22:25:25.750391 1884 consensus_queue.cc:260] T 3e58ce0ecc3e4295994c1d7dba61e156 P eb362cc342004cdc9d75207b00cbee35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } } peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } }
I20260811 22:25:25.743373 1889 tablet_bootstrap.cc:492] T 3e58ce0ecc3e4295994c1d7dba61e156 P a669f1e0074e4968bc0853b56071bdc0: Bootstrap starting.
I20260811 22:25:25.751691 850 catalog_manager.cc:4841] TS eb362cc342004cdc9d75207b00cbee35 (127.29.106.3:42937): tablet e26bf747b417404f925c0a8868323cc1 (table test-table [id=94fb2fa9bf3e44b5a82e1aaf3b39c627]) successfully deleted
I20260811 22:25:25.751811 1969 ts_tablet_manager.cc:1939] T e26bf747b417404f925c0a8868323cc1 P a669f1e0074e4968bc0853b56071bdc0: Deleting consensus metadata
I20260811 22:25:25.753000 1889 tablet_bootstrap.cc:654] T 3e58ce0ecc3e4295994c1d7dba61e156 P a669f1e0074e4968bc0853b56071bdc0: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:25.753587 847 catalog_manager.cc:4841] TS a669f1e0074e4968bc0853b56071bdc0 (127.29.106.2:37479): tablet e26bf747b417404f925c0a8868323cc1 (table test-table [id=94fb2fa9bf3e44b5a82e1aaf3b39c627]) successfully deleted
I20260811 22:25:25.753965 1884 ts_tablet_manager.cc:1428] T 3e58ce0ecc3e4295994c1d7dba61e156 P eb362cc342004cdc9d75207b00cbee35: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20260811 22:25:25.755481 1892 tablet_bootstrap.cc:492] T 3e58ce0ecc3e4295994c1d7dba61e156 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: No bootstrap required, opened a new log
I20260811 22:25:25.755661 1892 ts_tablet_manager.cc:1397] T 3e58ce0ecc3e4295994c1d7dba61e156 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.010s
I20260811 22:25:25.756384 1892 raft_consensus.cc:348] T 3e58ce0ecc3e4295994c1d7dba61e156 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } } peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } }
I20260811 22:25:25.756631 1892 raft_consensus.cc:374] T 3e58ce0ecc3e4295994c1d7dba61e156 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:25:25.756698 1892 raft_consensus.cc:729] T 3e58ce0ecc3e4295994c1d7dba61e156 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bb5c6c3715e4f7ebbcca58e9cf0e7f5, State: Initialized, Role: FOLLOWER
I20260811 22:25:25.756933 1892 consensus_queue.cc:260] T 3e58ce0ecc3e4295994c1d7dba61e156 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } } peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } }
I20260811 22:25:25.757690 1892 ts_tablet_manager.cc:1428] T 3e58ce0ecc3e4295994c1d7dba61e156 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260811 22:25:25.758976 1889 tablet_bootstrap.cc:492] T 3e58ce0ecc3e4295994c1d7dba61e156 P a669f1e0074e4968bc0853b56071bdc0: No bootstrap required, opened a new log
I20260811 22:25:25.759133 1889 ts_tablet_manager.cc:1397] T 3e58ce0ecc3e4295994c1d7dba61e156 P a669f1e0074e4968bc0853b56071bdc0: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.004s
I20260811 22:25:25.759891 1889 raft_consensus.cc:348] T 3e58ce0ecc3e4295994c1d7dba61e156 P a669f1e0074e4968bc0853b56071bdc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } } peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } }
I20260811 22:25:25.760144 1889 raft_consensus.cc:374] T 3e58ce0ecc3e4295994c1d7dba61e156 P a669f1e0074e4968bc0853b56071bdc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:25:25.760227 1889 raft_consensus.cc:729] T 3e58ce0ecc3e4295994c1d7dba61e156 P a669f1e0074e4968bc0853b56071bdc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a669f1e0074e4968bc0853b56071bdc0, State: Initialized, Role: FOLLOWER
I20260811 22:25:25.760439 1889 consensus_queue.cc:260] T 3e58ce0ecc3e4295994c1d7dba61e156 P a669f1e0074e4968bc0853b56071bdc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } } peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } }
I20260811 22:25:25.761108 1889 ts_tablet_manager.cc:1428] T 3e58ce0ecc3e4295994c1d7dba61e156 P a669f1e0074e4968bc0853b56071bdc0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260811 22:25:25.889838 1908 raft_consensus.cc:482] T 3e58ce0ecc3e4295994c1d7dba61e156 P eb362cc342004cdc9d75207b00cbee35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:25:25.890076 1908 raft_consensus.cc:504] T 3e58ce0ecc3e4295994c1d7dba61e156 P eb362cc342004cdc9d75207b00cbee35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } } peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } }
I20260811 22:25:25.890911 1908 leader_election.cc:288] T 3e58ce0ecc3e4295994c1d7dba61e156 P eb362cc342004cdc9d75207b00cbee35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 (127.29.106.1:33721), a669f1e0074e4968bc0853b56071bdc0 (127.29.106.2:37479)
I20260811 22:25:25.891497 1467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58ce0ecc3e4295994c1d7dba61e156" candidate_uuid: "eb362cc342004cdc9d75207b00cbee35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a669f1e0074e4968bc0853b56071bdc0" is_pre_election: true
I20260811 22:25:25.891486 1180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58ce0ecc3e4295994c1d7dba61e156" candidate_uuid: "eb362cc342004cdc9d75207b00cbee35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" is_pre_election: true
I20260811 22:25:25.891750 1467 raft_consensus.cc:2454] T 3e58ce0ecc3e4295994c1d7dba61e156 P a669f1e0074e4968bc0853b56071bdc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb362cc342004cdc9d75207b00cbee35 in term 0.
I20260811 22:25:25.891785 1180 raft_consensus.cc:2454] T 3e58ce0ecc3e4295994c1d7dba61e156 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb362cc342004cdc9d75207b00cbee35 in term 0.
I20260811 22:25:25.892308 1617 leader_election.cc:302] T 3e58ce0ecc3e4295994c1d7dba61e156 P eb362cc342004cdc9d75207b00cbee35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bb5c6c3715e4f7ebbcca58e9cf0e7f5, eb362cc342004cdc9d75207b00cbee35; no voters:
I20260811 22:25:25.892625 1908 raft_consensus.cc:2781] T 3e58ce0ecc3e4295994c1d7dba61e156 P eb362cc342004cdc9d75207b00cbee35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:25:25.892759 1908 raft_consensus.cc:482] T 3e58ce0ecc3e4295994c1d7dba61e156 P eb362cc342004cdc9d75207b00cbee35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:25:25.892858 1908 raft_consensus.cc:3037] T 3e58ce0ecc3e4295994c1d7dba61e156 P eb362cc342004cdc9d75207b00cbee35 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:25.896148 1908 raft_consensus.cc:504] T 3e58ce0ecc3e4295994c1d7dba61e156 P eb362cc342004cdc9d75207b00cbee35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } } peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } }
I20260811 22:25:25.896798 1908 leader_election.cc:288] T 3e58ce0ecc3e4295994c1d7dba61e156 P eb362cc342004cdc9d75207b00cbee35 [CANDIDATE]: Term 1 election: Requested vote from peers 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 (127.29.106.1:33721), a669f1e0074e4968bc0853b56071bdc0 (127.29.106.2:37479)
I20260811 22:25:25.897262 1180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58ce0ecc3e4295994c1d7dba61e156" candidate_uuid: "eb362cc342004cdc9d75207b00cbee35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5"
I20260811 22:25:25.897275 1467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e58ce0ecc3e4295994c1d7dba61e156" candidate_uuid: "eb362cc342004cdc9d75207b00cbee35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a669f1e0074e4968bc0853b56071bdc0"
I20260811 22:25:25.897502 1467 raft_consensus.cc:3037] T 3e58ce0ecc3e4295994c1d7dba61e156 P a669f1e0074e4968bc0853b56071bdc0 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:25.897518 1180 raft_consensus.cc:3037] T 3e58ce0ecc3e4295994c1d7dba61e156 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:25.900110 1467 raft_consensus.cc:2454] T 3e58ce0ecc3e4295994c1d7dba61e156 P a669f1e0074e4968bc0853b56071bdc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb362cc342004cdc9d75207b00cbee35 in term 1.
I20260811 22:25:25.900480 1180 raft_consensus.cc:2454] T 3e58ce0ecc3e4295994c1d7dba61e156 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb362cc342004cdc9d75207b00cbee35 in term 1.
I20260811 22:25:25.900516 1610 leader_election.cc:302] T 3e58ce0ecc3e4295994c1d7dba61e156 P eb362cc342004cdc9d75207b00cbee35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a669f1e0074e4968bc0853b56071bdc0, eb362cc342004cdc9d75207b00cbee35; no voters:
I20260811 22:25:25.900775 1908 raft_consensus.cc:2781] T 3e58ce0ecc3e4295994c1d7dba61e156 P eb362cc342004cdc9d75207b00cbee35 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:25:25.900934 1908 raft_consensus.cc:686] T 3e58ce0ecc3e4295994c1d7dba61e156 P eb362cc342004cdc9d75207b00cbee35 [term 1 LEADER]: Becoming Leader. State: Replica: eb362cc342004cdc9d75207b00cbee35, State: Running, Role: LEADER
I20260811 22:25:25.901196 1908 consensus_queue.cc:237] T 3e58ce0ecc3e4295994c1d7dba61e156 P eb362cc342004cdc9d75207b00cbee35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } } peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } }
I20260811 22:25:25.903965 876 catalog_manager.cc:5495] T 3e58ce0ecc3e4295994c1d7dba61e156 P eb362cc342004cdc9d75207b00cbee35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb362cc342004cdc9d75207b00cbee35 (127.29.106.3). New cstate: current_term: 1 leader_uuid: "eb362cc342004cdc9d75207b00cbee35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } health_report { overall_health: HEALTHY } } }
I20260811 22:25:25.955813 1180 raft_consensus.cc:1261] T 3e58ce0ecc3e4295994c1d7dba61e156 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [term 1 FOLLOWER]: Refusing update from remote peer eb362cc342004cdc9d75207b00cbee35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:25:25.955955 1467 raft_consensus.cc:1261] T 3e58ce0ecc3e4295994c1d7dba61e156 P a669f1e0074e4968bc0853b56071bdc0 [term 1 FOLLOWER]: Refusing update from remote peer eb362cc342004cdc9d75207b00cbee35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:25:25.956501 1915 consensus_queue.cc:1035] T 3e58ce0ecc3e4295994c1d7dba61e156 P eb362cc342004cdc9d75207b00cbee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:25:25.956760 1908 consensus_queue.cc:1035] T 3e58ce0ecc3e4295994c1d7dba61e156 P eb362cc342004cdc9d75207b00cbee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260811 22:25:25.980511 1944 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260811 22:25:27.580487 877 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:25:27.580849 877 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:25:27.586787 877 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc4020c4fbb7446aadc65b89aeb691ba: Sending DeleteTablet for 3 replicas of tablet 3e58ce0ecc3e4295994c1d7dba61e156
I20260811 22:25:27.587762 1427 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e58ce0ecc3e4295994c1d7dba61e156 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:25:27 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:39668
I20260811 22:25:27.587862 1706 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e58ce0ecc3e4295994c1d7dba61e156 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:25:27 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:37306
I20260811 22:25:27.588366 1132 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e58ce0ecc3e4295994c1d7dba61e156 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:25:27 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:55836
I20260811 22:25:27.588940 2021 tablet_replica.cc:330] T 3e58ce0ecc3e4295994c1d7dba61e156 P a669f1e0074e4968bc0853b56071bdc0: stopping tablet replica
I20260811 22:25:27.589210 2021 raft_consensus.cc:2229] T 3e58ce0ecc3e4295994c1d7dba61e156 P a669f1e0074e4968bc0853b56071bdc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:25:27.589740 2021 raft_consensus.cc:2258] T 3e58ce0ecc3e4295994c1d7dba61e156 P a669f1e0074e4968bc0853b56071bdc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:25:27.591252 2023 tablet_replica.cc:330] T 3e58ce0ecc3e4295994c1d7dba61e156 P eb362cc342004cdc9d75207b00cbee35: stopping tablet replica
I20260811 22:25:27.591614 2023 raft_consensus.cc:2229] T 3e58ce0ecc3e4295994c1d7dba61e156 P eb362cc342004cdc9d75207b00cbee35 [term 1 LEADER]: Raft consensus shutting down.
I20260811 22:25:27.591763 2021 ts_tablet_manager.cc:1905] T 3e58ce0ecc3e4295994c1d7dba61e156 P a669f1e0074e4968bc0853b56071bdc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:25:27.592335 2023 raft_consensus.cc:2258] T 3e58ce0ecc3e4295994c1d7dba61e156 P eb362cc342004cdc9d75207b00cbee35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:25:27.593442 2023 ts_tablet_manager.cc:1905] T 3e58ce0ecc3e4295994c1d7dba61e156 P eb362cc342004cdc9d75207b00cbee35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:25:27.599318 2024 tablet_replica.cc:330] T 3e58ce0ecc3e4295994c1d7dba61e156 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: stopping tablet replica
I20260811 22:25:27.599721 2024 raft_consensus.cc:2229] T 3e58ce0ecc3e4295994c1d7dba61e156 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:25:27.600219 2024 raft_consensus.cc:2258] T 3e58ce0ecc3e4295994c1d7dba61e156 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:25:27.600658 2021 ts_tablet_manager.cc:1918] T 3e58ce0ecc3e4295994c1d7dba61e156 P a669f1e0074e4968bc0853b56071bdc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260811 22:25:27.600898 2021 log.cc:1196] T 3e58ce0ecc3e4295994c1d7dba61e156 P a669f1e0074e4968bc0853b56071bdc0: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-1/wal/wals/3e58ce0ecc3e4295994c1d7dba61e156
I20260811 22:25:27.601329 2024 ts_tablet_manager.cc:1905] T 3e58ce0ecc3e4295994c1d7dba61e156 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:25:27.601493 2021 ts_tablet_manager.cc:1939] T 3e58ce0ecc3e4295994c1d7dba61e156 P a669f1e0074e4968bc0853b56071bdc0: Deleting consensus metadata
I20260811 22:25:27.602584 847 catalog_manager.cc:4841] TS a669f1e0074e4968bc0853b56071bdc0 (127.29.106.2:37479): tablet 3e58ce0ecc3e4295994c1d7dba61e156 (table test-table [id=62baec03d271427995cbd4129f33a1f1]) successfully deleted
I20260811 22:25:27.603332 2023 ts_tablet_manager.cc:1918] T 3e58ce0ecc3e4295994c1d7dba61e156 P eb362cc342004cdc9d75207b00cbee35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260811 22:25:27.603582 2023 log.cc:1196] T 3e58ce0ecc3e4295994c1d7dba61e156 P eb362cc342004cdc9d75207b00cbee35: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-2/wal/wals/3e58ce0ecc3e4295994c1d7dba61e156
I20260811 22:25:27.604538 2023 ts_tablet_manager.cc:1939] T 3e58ce0ecc3e4295994c1d7dba61e156 P eb362cc342004cdc9d75207b00cbee35: Deleting consensus metadata
I20260811 22:25:27.605669 850 catalog_manager.cc:4841] TS eb362cc342004cdc9d75207b00cbee35 (127.29.106.3:42937): tablet 3e58ce0ecc3e4295994c1d7dba61e156 (table test-table [id=62baec03d271427995cbd4129f33a1f1]) successfully deleted
I20260811 22:25:27.609068 2024 ts_tablet_manager.cc:1918] T 3e58ce0ecc3e4295994c1d7dba61e156 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260811 22:25:27.609218 2024 log.cc:1196] T 3e58ce0ecc3e4295994c1d7dba61e156 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-0/wal/wals/3e58ce0ecc3e4295994c1d7dba61e156
I20260811 22:25:27.609791 2024 ts_tablet_manager.cc:1939] T 3e58ce0ecc3e4295994c1d7dba61e156 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: Deleting consensus metadata
I20260811 22:25:27.610764 850 catalog_manager.cc:4841] TS 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 (127.29.106.1:33721): tablet 3e58ce0ecc3e4295994c1d7dba61e156 (table test-table [id=62baec03d271427995cbd4129f33a1f1]) successfully deleted
Aug 11 22:25:27 dist-test-slave-gjrz krb5kdc[778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487127, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 11 22:25:27 dist-test-slave-gjrz krb5kdc[778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487127, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:25:27.652138 2035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:27.630897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40162 (local address 127.29.106.62:40297)
0811 22:25:27.631615 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:27.631622 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:27.631786 (+ 164us) server_negotiation.cc:407] Connection header received
0811 22:25:27.631942 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:27.631945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:27.632060 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:27.632148 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:27.634223 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:27.635113 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:27.637163 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:27.637926 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:27.638065 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:27.645787 (+ 7722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:27.645794 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:27.645797 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:27.645829 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:27.649579 (+ 3750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:27.650142 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:27.650146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:27.650148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:27.650303 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:27.650591 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:27.650594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:27.650596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:27.650947 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:27.651202 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:27.651375 (+ 173us) server_negotiation.cc:299] Negotiation successful
0811 22:25:27.651582 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":375,"threads_started":1}
I20260811 22:25:27.655823 877 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40162:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:25:27.656472 877 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:25:27.663341 1132 tablet_service.cc:1467] Processing CreateTablet for tablet 9188e1c05f234542aa1480d7862c2ebf (DEFAULT_TABLE table=test-table [id=9e4f1e2dbaae4494b94efd6d1814d325]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:25:27.663333 1427 tablet_service.cc:1467] Processing CreateTablet for tablet 9188e1c05f234542aa1480d7862c2ebf (DEFAULT_TABLE table=test-table [id=9e4f1e2dbaae4494b94efd6d1814d325]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:25:27.663509 1706 tablet_service.cc:1467] Processing CreateTablet for tablet 9188e1c05f234542aa1480d7862c2ebf (DEFAULT_TABLE table=test-table [id=9e4f1e2dbaae4494b94efd6d1814d325]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:25:27.663779 1427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9188e1c05f234542aa1480d7862c2ebf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:27.663930 1132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9188e1c05f234542aa1480d7862c2ebf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:27.664072 1706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9188e1c05f234542aa1480d7862c2ebf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:27.671944 2037 tablet_bootstrap.cc:492] T 9188e1c05f234542aa1480d7862c2ebf P eb362cc342004cdc9d75207b00cbee35: Bootstrap starting.
I20260811 22:25:27.672724 2040 tablet_bootstrap.cc:492] T 9188e1c05f234542aa1480d7862c2ebf P a669f1e0074e4968bc0853b56071bdc0: Bootstrap starting.
I20260811 22:25:27.673890 2041 tablet_bootstrap.cc:492] T 9188e1c05f234542aa1480d7862c2ebf P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: Bootstrap starting.
I20260811 22:25:27.676194 2040 tablet_bootstrap.cc:654] T 9188e1c05f234542aa1480d7862c2ebf P a669f1e0074e4968bc0853b56071bdc0: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:27.676548 2041 tablet_bootstrap.cc:654] T 9188e1c05f234542aa1480d7862c2ebf P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:27.676715 2037 tablet_bootstrap.cc:654] T 9188e1c05f234542aa1480d7862c2ebf P eb362cc342004cdc9d75207b00cbee35: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:27.679951 2041 tablet_bootstrap.cc:492] T 9188e1c05f234542aa1480d7862c2ebf P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: No bootstrap required, opened a new log
I20260811 22:25:27.680150 2041 ts_tablet_manager.cc:1397] T 9188e1c05f234542aa1480d7862c2ebf P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260811 22:25:27.681326 2037 tablet_bootstrap.cc:492] T 9188e1c05f234542aa1480d7862c2ebf P eb362cc342004cdc9d75207b00cbee35: No bootstrap required, opened a new log
I20260811 22:25:27.681380 2040 tablet_bootstrap.cc:492] T 9188e1c05f234542aa1480d7862c2ebf P a669f1e0074e4968bc0853b56071bdc0: No bootstrap required, opened a new log
I20260811 22:25:27.681505 2037 ts_tablet_manager.cc:1397] T 9188e1c05f234542aa1480d7862c2ebf P eb362cc342004cdc9d75207b00cbee35: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260811 22:25:27.681528 2040 ts_tablet_manager.cc:1397] T 9188e1c05f234542aa1480d7862c2ebf P a669f1e0074e4968bc0853b56071bdc0: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260811 22:25:27.681283 2041 raft_consensus.cc:348] T 9188e1c05f234542aa1480d7862c2ebf P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } } peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } }
I20260811 22:25:27.681664 2041 raft_consensus.cc:374] T 9188e1c05f234542aa1480d7862c2ebf P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:25:27.681831 2041 raft_consensus.cc:729] T 9188e1c05f234542aa1480d7862c2ebf P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bb5c6c3715e4f7ebbcca58e9cf0e7f5, State: Initialized, Role: FOLLOWER
I20260811 22:25:27.682140 2041 consensus_queue.cc:260] T 9188e1c05f234542aa1480d7862c2ebf P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } } peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } }
I20260811 22:25:27.682578 2040 raft_consensus.cc:348] T 9188e1c05f234542aa1480d7862c2ebf P a669f1e0074e4968bc0853b56071bdc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } } peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } }
I20260811 22:25:27.682940 2040 raft_consensus.cc:374] T 9188e1c05f234542aa1480d7862c2ebf P a669f1e0074e4968bc0853b56071bdc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:25:27.682758 2037 raft_consensus.cc:348] T 9188e1c05f234542aa1480d7862c2ebf P eb362cc342004cdc9d75207b00cbee35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } } peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } }
I20260811 22:25:27.683111 2037 raft_consensus.cc:374] T 9188e1c05f234542aa1480d7862c2ebf P eb362cc342004cdc9d75207b00cbee35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:25:27.683138 2040 raft_consensus.cc:729] T 9188e1c05f234542aa1480d7862c2ebf P a669f1e0074e4968bc0853b56071bdc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a669f1e0074e4968bc0853b56071bdc0, State: Initialized, Role: FOLLOWER
I20260811 22:25:27.683364 2037 raft_consensus.cc:729] T 9188e1c05f234542aa1480d7862c2ebf P eb362cc342004cdc9d75207b00cbee35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb362cc342004cdc9d75207b00cbee35, State: Initialized, Role: FOLLOWER
I20260811 22:25:27.683521 2040 consensus_queue.cc:260] T 9188e1c05f234542aa1480d7862c2ebf P a669f1e0074e4968bc0853b56071bdc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } } peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } }
I20260811 22:25:27.683697 2041 ts_tablet_manager.cc:1428] T 9188e1c05f234542aa1480d7862c2ebf P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260811 22:25:27.683799 2037 consensus_queue.cc:260] T 9188e1c05f234542aa1480d7862c2ebf P eb362cc342004cdc9d75207b00cbee35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } } peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } }
I20260811 22:25:27.685420 2037 ts_tablet_manager.cc:1428] T 9188e1c05f234542aa1480d7862c2ebf P eb362cc342004cdc9d75207b00cbee35: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260811 22:25:27.685636 2040 ts_tablet_manager.cc:1428] T 9188e1c05f234542aa1480d7862c2ebf P a669f1e0074e4968bc0853b56071bdc0: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260811 22:25:27.761967 1999 raft_consensus.cc:482] T 9188e1c05f234542aa1480d7862c2ebf P eb362cc342004cdc9d75207b00cbee35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:25:27.762212 1999 raft_consensus.cc:504] T 9188e1c05f234542aa1480d7862c2ebf P eb362cc342004cdc9d75207b00cbee35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } } peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } }
I20260811 22:25:27.763075 1999 leader_election.cc:288] T 9188e1c05f234542aa1480d7862c2ebf P eb362cc342004cdc9d75207b00cbee35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 (127.29.106.1:33721), a669f1e0074e4968bc0853b56071bdc0 (127.29.106.2:37479)
I20260811 22:25:27.763757 1467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9188e1c05f234542aa1480d7862c2ebf" candidate_uuid: "eb362cc342004cdc9d75207b00cbee35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a669f1e0074e4968bc0853b56071bdc0" is_pre_election: true
I20260811 22:25:27.763727 1180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9188e1c05f234542aa1480d7862c2ebf" candidate_uuid: "eb362cc342004cdc9d75207b00cbee35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" is_pre_election: true
I20260811 22:25:27.764051 1467 raft_consensus.cc:2454] T 9188e1c05f234542aa1480d7862c2ebf P a669f1e0074e4968bc0853b56071bdc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb362cc342004cdc9d75207b00cbee35 in term 0.
I20260811 22:25:27.764308 1180 raft_consensus.cc:2454] T 9188e1c05f234542aa1480d7862c2ebf P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb362cc342004cdc9d75207b00cbee35 in term 0.
I20260811 22:25:27.764659 1610 leader_election.cc:302] T 9188e1c05f234542aa1480d7862c2ebf P eb362cc342004cdc9d75207b00cbee35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a669f1e0074e4968bc0853b56071bdc0, eb362cc342004cdc9d75207b00cbee35; no voters:
I20260811 22:25:27.764983 1999 raft_consensus.cc:2781] T 9188e1c05f234542aa1480d7862c2ebf P eb362cc342004cdc9d75207b00cbee35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:25:27.765228 1999 raft_consensus.cc:482] T 9188e1c05f234542aa1480d7862c2ebf P eb362cc342004cdc9d75207b00cbee35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:25:27.765470 1999 raft_consensus.cc:3037] T 9188e1c05f234542aa1480d7862c2ebf P eb362cc342004cdc9d75207b00cbee35 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:27.768606 1999 raft_consensus.cc:504] T 9188e1c05f234542aa1480d7862c2ebf P eb362cc342004cdc9d75207b00cbee35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } } peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } }
I20260811 22:25:27.769248 1999 leader_election.cc:288] T 9188e1c05f234542aa1480d7862c2ebf P eb362cc342004cdc9d75207b00cbee35 [CANDIDATE]: Term 1 election: Requested vote from peers 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 (127.29.106.1:33721), a669f1e0074e4968bc0853b56071bdc0 (127.29.106.2:37479)
I20260811 22:25:27.769695 1180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9188e1c05f234542aa1480d7862c2ebf" candidate_uuid: "eb362cc342004cdc9d75207b00cbee35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5"
I20260811 22:25:27.769868 1467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9188e1c05f234542aa1480d7862c2ebf" candidate_uuid: "eb362cc342004cdc9d75207b00cbee35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a669f1e0074e4968bc0853b56071bdc0"
I20260811 22:25:27.770020 1180 raft_consensus.cc:3037] T 9188e1c05f234542aa1480d7862c2ebf P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:27.770159 1467 raft_consensus.cc:3037] T 9188e1c05f234542aa1480d7862c2ebf P a669f1e0074e4968bc0853b56071bdc0 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:27.772742 1180 raft_consensus.cc:2454] T 9188e1c05f234542aa1480d7862c2ebf P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb362cc342004cdc9d75207b00cbee35 in term 1.
I20260811 22:25:27.772981 1467 raft_consensus.cc:2454] T 9188e1c05f234542aa1480d7862c2ebf P a669f1e0074e4968bc0853b56071bdc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb362cc342004cdc9d75207b00cbee35 in term 1.
I20260811 22:25:27.773277 1617 leader_election.cc:302] T 9188e1c05f234542aa1480d7862c2ebf P eb362cc342004cdc9d75207b00cbee35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bb5c6c3715e4f7ebbcca58e9cf0e7f5, eb362cc342004cdc9d75207b00cbee35; no voters:
I20260811 22:25:27.773753 1999 raft_consensus.cc:2781] T 9188e1c05f234542aa1480d7862c2ebf P eb362cc342004cdc9d75207b00cbee35 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:25:27.774020 1999 raft_consensus.cc:686] T 9188e1c05f234542aa1480d7862c2ebf P eb362cc342004cdc9d75207b00cbee35 [term 1 LEADER]: Becoming Leader. State: Replica: eb362cc342004cdc9d75207b00cbee35, State: Running, Role: LEADER
I20260811 22:25:27.774382 1999 consensus_queue.cc:237] T 9188e1c05f234542aa1480d7862c2ebf P eb362cc342004cdc9d75207b00cbee35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } } peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } }
I20260811 22:25:27.777845 876 catalog_manager.cc:5495] T 9188e1c05f234542aa1480d7862c2ebf P eb362cc342004cdc9d75207b00cbee35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb362cc342004cdc9d75207b00cbee35 (127.29.106.3). New cstate: current_term: 1 leader_uuid: "eb362cc342004cdc9d75207b00cbee35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } health_report { overall_health: HEALTHY } } }
I20260811 22:25:27.810297 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:27.803288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37324 (local address 127.29.106.3:42937)
0811 22:25:27.804264 (+ 976us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:27.804269 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:27.804308 (+ 39us) server_negotiation.cc:407] Connection header received
0811 22:25:27.804358 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:27.804361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:27.804421 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:27.804498 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:25:27.804952 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:27.806197 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:27.807759 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:27.808435 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:27.808549 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:27.809197 (+ 648us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:25:27.809298 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:25:27.809375 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:27.809578 (+ 203us) server_negotiation.cc:299] Negotiation successful
0811 22:25:27.809658 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":827,"thread_start_us":637,"threads_started":1}
I20260811 22:25:27.820276 1467 raft_consensus.cc:1261] T 9188e1c05f234542aa1480d7862c2ebf P a669f1e0074e4968bc0853b56071bdc0 [term 1 FOLLOWER]: Refusing update from remote peer eb362cc342004cdc9d75207b00cbee35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:25:27.820374 1180 raft_consensus.cc:1261] T 9188e1c05f234542aa1480d7862c2ebf P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [term 1 FOLLOWER]: Refusing update from remote peer eb362cc342004cdc9d75207b00cbee35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:25:27.820868 1955 consensus_queue.cc:1035] T 9188e1c05f234542aa1480d7862c2ebf P eb362cc342004cdc9d75207b00cbee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:25:27.821456 1986 consensus_queue.cc:1035] T 9188e1c05f234542aa1480d7862c2ebf P eb362cc342004cdc9d75207b00cbee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:25:27.868199 877 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:25:27.868481 877 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:25:27.871548 877 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc4020c4fbb7446aadc65b89aeb691ba: Sending DeleteTablet for 3 replicas of tablet 9188e1c05f234542aa1480d7862c2ebf
I20260811 22:25:27.872336 1132 tablet_service.cc:1514] Processing DeleteTablet for tablet 9188e1c05f234542aa1480d7862c2ebf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:25:27 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:55836
I20260811 22:25:27.872615 1427 tablet_service.cc:1514] Processing DeleteTablet for tablet 9188e1c05f234542aa1480d7862c2ebf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:25:27 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:39668
I20260811 22:25:27.872702 2024 tablet_replica.cc:330] T 9188e1c05f234542aa1480d7862c2ebf P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: stopping tablet replica
I20260811 22:25:27.872699 1706 tablet_service.cc:1514] Processing DeleteTablet for tablet 9188e1c05f234542aa1480d7862c2ebf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:25:27 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:37306
I20260811 22:25:27.872997 2024 raft_consensus.cc:2229] T 9188e1c05f234542aa1480d7862c2ebf P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:25:27.873080 2023 tablet_replica.cc:330] T 9188e1c05f234542aa1480d7862c2ebf P eb362cc342004cdc9d75207b00cbee35: stopping tablet replica
I20260811 22:25:27.873101 2021 tablet_replica.cc:330] T 9188e1c05f234542aa1480d7862c2ebf P a669f1e0074e4968bc0853b56071bdc0: stopping tablet replica
I20260811 22:25:27.873322 2024 raft_consensus.cc:2258] T 9188e1c05f234542aa1480d7862c2ebf P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:25:27.873376 2023 raft_consensus.cc:2229] T 9188e1c05f234542aa1480d7862c2ebf P eb362cc342004cdc9d75207b00cbee35 [term 1 LEADER]: Raft consensus shutting down.
I20260811 22:25:27.873417 2021 raft_consensus.cc:2229] T 9188e1c05f234542aa1480d7862c2ebf P a669f1e0074e4968bc0853b56071bdc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:25:27.873741 2021 raft_consensus.cc:2258] T 9188e1c05f234542aa1480d7862c2ebf P a669f1e0074e4968bc0853b56071bdc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:25:27.873441 877 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40162:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260811 22:25:27.873745 2023 raft_consensus.cc:2258] T 9188e1c05f234542aa1480d7862c2ebf P eb362cc342004cdc9d75207b00cbee35 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260811 22:25:27.874384 877 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:25:27.874655 2023 ts_tablet_manager.cc:1905] T 9188e1c05f234542aa1480d7862c2ebf P eb362cc342004cdc9d75207b00cbee35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:25:27.874655 2024 ts_tablet_manager.cc:1905] T 9188e1c05f234542aa1480d7862c2ebf P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:25:27.874945 2021 ts_tablet_manager.cc:1905] T 9188e1c05f234542aa1480d7862c2ebf P a669f1e0074e4968bc0853b56071bdc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:25:27.882836 2024 ts_tablet_manager.cc:1918] T 9188e1c05f234542aa1480d7862c2ebf P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 22:25:27.883015 1427 tablet_service.cc:1467] Processing CreateTablet for tablet 34e2c1c03e394ae99687e823a70da469 (DEFAULT_TABLE table=test-table [id=5fa4711085074120b51c44fe6a0887a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:25:27.883286 2024 log.cc:1196] T 9188e1c05f234542aa1480d7862c2ebf P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-0/wal/wals/9188e1c05f234542aa1480d7862c2ebf
I20260811 22:25:27.883626 1427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34e2c1c03e394ae99687e823a70da469. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:27.883915 2021 ts_tablet_manager.cc:1918] T 9188e1c05f234542aa1480d7862c2ebf P a669f1e0074e4968bc0853b56071bdc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 22:25:27.884125 2023 ts_tablet_manager.cc:1918] T 9188e1c05f234542aa1480d7862c2ebf P eb362cc342004cdc9d75207b00cbee35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 22:25:27.884214 2024 ts_tablet_manager.cc:1939] T 9188e1c05f234542aa1480d7862c2ebf P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: Deleting consensus metadata
I20260811 22:25:27.884284 2021 log.cc:1196] T 9188e1c05f234542aa1480d7862c2ebf P a669f1e0074e4968bc0853b56071bdc0: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-1/wal/wals/9188e1c05f234542aa1480d7862c2ebf
I20260811 22:25:27.884416 2023 log.cc:1196] T 9188e1c05f234542aa1480d7862c2ebf P eb362cc342004cdc9d75207b00cbee35: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-2/wal/wals/9188e1c05f234542aa1480d7862c2ebf
I20260811 22:25:27.885095 2021 ts_tablet_manager.cc:1939] T 9188e1c05f234542aa1480d7862c2ebf P a669f1e0074e4968bc0853b56071bdc0: Deleting consensus metadata
I20260811 22:25:27.885272 2023 ts_tablet_manager.cc:1939] T 9188e1c05f234542aa1480d7862c2ebf P eb362cc342004cdc9d75207b00cbee35: Deleting consensus metadata
I20260811 22:25:27.886251 847 catalog_manager.cc:4841] TS a669f1e0074e4968bc0853b56071bdc0 (127.29.106.2:37479): tablet 9188e1c05f234542aa1480d7862c2ebf (table test-table [id=9e4f1e2dbaae4494b94efd6d1814d325]) successfully deleted
I20260811 22:25:27.886341 850 catalog_manager.cc:4841] TS 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 (127.29.106.1:33721): tablet 9188e1c05f234542aa1480d7862c2ebf (table test-table [id=9e4f1e2dbaae4494b94efd6d1814d325]) successfully deleted
I20260811 22:25:27.886703 850 catalog_manager.cc:4841] TS eb362cc342004cdc9d75207b00cbee35 (127.29.106.3:42937): tablet 9188e1c05f234542aa1480d7862c2ebf (table test-table [id=9e4f1e2dbaae4494b94efd6d1814d325]) successfully deleted
I20260811 22:25:27.887042 1706 tablet_service.cc:1467] Processing CreateTablet for tablet 34e2c1c03e394ae99687e823a70da469 (DEFAULT_TABLE table=test-table [id=5fa4711085074120b51c44fe6a0887a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:25:27.887437 1706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34e2c1c03e394ae99687e823a70da469. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:27.887471 1132 tablet_service.cc:1467] Processing CreateTablet for tablet 34e2c1c03e394ae99687e823a70da469 (DEFAULT_TABLE table=test-table [id=5fa4711085074120b51c44fe6a0887a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:25:27.887967 1132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34e2c1c03e394ae99687e823a70da469. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:27.892026 2040 tablet_bootstrap.cc:492] T 34e2c1c03e394ae99687e823a70da469 P a669f1e0074e4968bc0853b56071bdc0: Bootstrap starting.
I20260811 22:25:27.895242 2040 tablet_bootstrap.cc:654] T 34e2c1c03e394ae99687e823a70da469 P a669f1e0074e4968bc0853b56071bdc0: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:27.895602 2037 tablet_bootstrap.cc:492] T 34e2c1c03e394ae99687e823a70da469 P eb362cc342004cdc9d75207b00cbee35: Bootstrap starting.
I20260811 22:25:27.895982 2041 tablet_bootstrap.cc:492] T 34e2c1c03e394ae99687e823a70da469 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: Bootstrap starting.
I20260811 22:25:27.898401 2037 tablet_bootstrap.cc:654] T 34e2c1c03e394ae99687e823a70da469 P eb362cc342004cdc9d75207b00cbee35: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:27.898707 2040 tablet_bootstrap.cc:492] T 34e2c1c03e394ae99687e823a70da469 P a669f1e0074e4968bc0853b56071bdc0: No bootstrap required, opened a new log
I20260811 22:25:27.898789 2041 tablet_bootstrap.cc:654] T 34e2c1c03e394ae99687e823a70da469 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:27.898958 2040 ts_tablet_manager.cc:1397] T 34e2c1c03e394ae99687e823a70da469 P a669f1e0074e4968bc0853b56071bdc0: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20260811 22:25:27.900017 2040 raft_consensus.cc:348] T 34e2c1c03e394ae99687e823a70da469 P a669f1e0074e4968bc0853b56071bdc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } } peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } }
I20260811 22:25:27.900467 2040 raft_consensus.cc:374] T 34e2c1c03e394ae99687e823a70da469 P a669f1e0074e4968bc0853b56071bdc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:25:27.900691 2040 raft_consensus.cc:729] T 34e2c1c03e394ae99687e823a70da469 P a669f1e0074e4968bc0853b56071bdc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a669f1e0074e4968bc0853b56071bdc0, State: Initialized, Role: FOLLOWER
I20260811 22:25:27.901149 2040 consensus_queue.cc:260] T 34e2c1c03e394ae99687e823a70da469 P a669f1e0074e4968bc0853b56071bdc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } } peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } }
I20260811 22:25:27.901991 2037 tablet_bootstrap.cc:492] T 34e2c1c03e394ae99687e823a70da469 P eb362cc342004cdc9d75207b00cbee35: No bootstrap required, opened a new log
I20260811 22:25:27.902313 2037 ts_tablet_manager.cc:1397] T 34e2c1c03e394ae99687e823a70da469 P eb362cc342004cdc9d75207b00cbee35: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260811 22:25:27.902920 2040 ts_tablet_manager.cc:1428] T 34e2c1c03e394ae99687e823a70da469 P a669f1e0074e4968bc0853b56071bdc0: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260811 22:25:27.904698 2041 tablet_bootstrap.cc:492] T 34e2c1c03e394ae99687e823a70da469 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: No bootstrap required, opened a new log
I20260811 22:25:27.904534 2037 raft_consensus.cc:348] T 34e2c1c03e394ae99687e823a70da469 P eb362cc342004cdc9d75207b00cbee35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } } peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } }
I20260811 22:25:27.904913 2041 ts_tablet_manager.cc:1397] T 34e2c1c03e394ae99687e823a70da469 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260811 22:25:27.905052 2037 raft_consensus.cc:374] T 34e2c1c03e394ae99687e823a70da469 P eb362cc342004cdc9d75207b00cbee35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:25:27.905211 2037 raft_consensus.cc:729] T 34e2c1c03e394ae99687e823a70da469 P eb362cc342004cdc9d75207b00cbee35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb362cc342004cdc9d75207b00cbee35, State: Initialized, Role: FOLLOWER
I20260811 22:25:27.905573 2037 consensus_queue.cc:260] T 34e2c1c03e394ae99687e823a70da469 P eb362cc342004cdc9d75207b00cbee35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } } peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } }
I20260811 22:25:27.905849 2041 raft_consensus.cc:348] T 34e2c1c03e394ae99687e823a70da469 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } } peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } }
I20260811 22:25:27.906286 2041 raft_consensus.cc:374] T 34e2c1c03e394ae99687e823a70da469 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:25:27.906315 2037 ts_tablet_manager.cc:1428] T 34e2c1c03e394ae99687e823a70da469 P eb362cc342004cdc9d75207b00cbee35: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20260811 22:25:27.906450 2041 raft_consensus.cc:729] T 34e2c1c03e394ae99687e823a70da469 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bb5c6c3715e4f7ebbcca58e9cf0e7f5, State: Initialized, Role: FOLLOWER
I20260811 22:25:27.906837 2041 consensus_queue.cc:260] T 34e2c1c03e394ae99687e823a70da469 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } } peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } }
I20260811 22:25:27.907608 2041 ts_tablet_manager.cc:1428] T 34e2c1c03e394ae99687e823a70da469 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260811 22:25:28.047820 2046 raft_consensus.cc:482] T 34e2c1c03e394ae99687e823a70da469 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:25:28.048223 2046 raft_consensus.cc:504] T 34e2c1c03e394ae99687e823a70da469 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } } peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } }
I20260811 22:25:28.050443 2046 leader_election.cc:288] T 34e2c1c03e394ae99687e823a70da469 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a669f1e0074e4968bc0853b56071bdc0 (127.29.106.2:37479), eb362cc342004cdc9d75207b00cbee35 (127.29.106.3:42937)
I20260811 22:25:28.070156 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:28.051439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:38985 (local address 127.29.106.3:42937)
0811 22:25:28.051625 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:28.051632 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:28.060278 (+ 8646us) server_negotiation.cc:407] Connection header received
0811 22:25:28.060462 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:28.060465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:28.060518 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:28.060603 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:25:28.061968 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:28.063381 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:28.066983 (+ 3602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:28.068831 (+ 1848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:28.068983 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:28.069057 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:28.069719 (+ 662us) server_negotiation.cc:299] Negotiation successful
0811 22:25:28.069838 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260811 22:25:28.070266 2072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:28.051283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:39857 (local address 127.29.106.2:37479)
0811 22:25:28.058167 (+ 6884us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:28.058174 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:28.059787 (+ 1613us) server_negotiation.cc:407] Connection header received
0811 22:25:28.059869 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:28.059874 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:28.059949 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:28.060031 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:25:28.061294 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:28.063026 (+ 1732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:28.066866 (+ 3840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:28.068771 (+ 1905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:28.069017 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:28.069095 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:28.069601 (+ 506us) server_negotiation.cc:299] Negotiation successful
0811 22:25:28.069736 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6716,"thread_start_us":6408,"threads_started":1}
I20260811 22:25:28.070641 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:28.050580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.2:37479 (local address 127.29.106.1:39857)
0811 22:25:28.059427 (+ 8847us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:28.059451 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:28.059657 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:28.060089 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:28.060094 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:28.060121 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:25:28.061017 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:28.061056 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:28.063244 (+ 2188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:28.063248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:28.066726 (+ 3478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:28.066741 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:28.069178 (+ 2437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:28.069187 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:28.069419 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:28.069458 (+ 39us) client_negotiation.cc:769] Sending connection context
0811 22:25:28.069584 (+ 126us) client_negotiation.cc:240] Negotiation successful
0811 22:25:28.069724 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8580,"mutex_wait_us":25,"spinlock_wait_cycles":125824,"thread_start_us":9202,"threads_started":1}
I20260811 22:25:28.070760 2070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:28.051041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.3:42937 (local address 127.29.106.1:38985)
0811 22:25:28.060154 (+ 9113us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:28.060171 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:28.060346 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:28.060634 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:28.060638 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:28.060661 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:25:28.061721 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:28.061734 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:28.063511 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:28.063516 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:28.066858 (+ 3342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:28.066871 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:28.069322 (+ 2451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:28.069330 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:28.069559 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:28.069590 (+ 31us) client_negotiation.cc:769] Sending connection context
0811 22:25:28.069663 (+ 73us) client_negotiation.cc:240] Negotiation successful
0811 22:25:28.069768 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8956,"mutex_wait_us":4,"thread_start_us":8209,"threads_started":1}
I20260811 22:25:28.071640 1468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34e2c1c03e394ae99687e823a70da469" candidate_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a669f1e0074e4968bc0853b56071bdc0" is_pre_election: true
I20260811 22:25:28.071888 1468 raft_consensus.cc:2454] T 34e2c1c03e394ae99687e823a70da469 P a669f1e0074e4968bc0853b56071bdc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 in term 0.
I20260811 22:25:28.071821 1761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34e2c1c03e394ae99687e823a70da469" candidate_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb362cc342004cdc9d75207b00cbee35" is_pre_election: true
I20260811 22:25:28.072402 1048 leader_election.cc:302] T 34e2c1c03e394ae99687e823a70da469 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bb5c6c3715e4f7ebbcca58e9cf0e7f5, a669f1e0074e4968bc0853b56071bdc0; no voters:
I20260811 22:25:28.072502 1761 raft_consensus.cc:2454] T 34e2c1c03e394ae99687e823a70da469 P eb362cc342004cdc9d75207b00cbee35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 in term 0.
I20260811 22:25:28.073058 2046 raft_consensus.cc:2781] T 34e2c1c03e394ae99687e823a70da469 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:25:28.073325 2046 raft_consensus.cc:482] T 34e2c1c03e394ae99687e823a70da469 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:25:28.073529 2046 raft_consensus.cc:3037] T 34e2c1c03e394ae99687e823a70da469 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:28.076907 2046 raft_consensus.cc:504] T 34e2c1c03e394ae99687e823a70da469 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } } peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } }
I20260811 22:25:28.077883 2046 leader_election.cc:288] T 34e2c1c03e394ae99687e823a70da469 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [CANDIDATE]: Term 1 election: Requested vote from peers a669f1e0074e4968bc0853b56071bdc0 (127.29.106.2:37479), eb362cc342004cdc9d75207b00cbee35 (127.29.106.3:42937)
I20260811 22:25:28.078446 1468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34e2c1c03e394ae99687e823a70da469" candidate_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a669f1e0074e4968bc0853b56071bdc0"
I20260811 22:25:28.078537 1761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34e2c1c03e394ae99687e823a70da469" candidate_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb362cc342004cdc9d75207b00cbee35"
I20260811 22:25:28.078737 1468 raft_consensus.cc:3037] T 34e2c1c03e394ae99687e823a70da469 P a669f1e0074e4968bc0853b56071bdc0 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:28.078846 1761 raft_consensus.cc:3037] T 34e2c1c03e394ae99687e823a70da469 P eb362cc342004cdc9d75207b00cbee35 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:28.082857 1468 raft_consensus.cc:2454] T 34e2c1c03e394ae99687e823a70da469 P a669f1e0074e4968bc0853b56071bdc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 in term 1.
I20260811 22:25:28.083328 1048 leader_election.cc:302] T 34e2c1c03e394ae99687e823a70da469 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bb5c6c3715e4f7ebbcca58e9cf0e7f5, a669f1e0074e4968bc0853b56071bdc0; no voters:
I20260811 22:25:28.083401 1761 raft_consensus.cc:2454] T 34e2c1c03e394ae99687e823a70da469 P eb362cc342004cdc9d75207b00cbee35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 in term 1.
I20260811 22:25:28.083734 2046 raft_consensus.cc:2781] T 34e2c1c03e394ae99687e823a70da469 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:25:28.084581 2046 raft_consensus.cc:686] T 34e2c1c03e394ae99687e823a70da469 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [term 1 LEADER]: Becoming Leader. State: Replica: 2bb5c6c3715e4f7ebbcca58e9cf0e7f5, State: Running, Role: LEADER
I20260811 22:25:28.085116 2046 consensus_queue.cc:237] T 34e2c1c03e394ae99687e823a70da469 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } } peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } }
I20260811 22:25:28.089340 876 catalog_manager.cc:5495] T 34e2c1c03e394ae99687e823a70da469 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 (127.29.106.1). New cstate: current_term: 1 leader_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb5c6c3715e4f7ebbcca58e9cf0e7f5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 } health_report { overall_health: UNKNOWN } } }
I20260811 22:25:28.151326 2076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:28.142695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55842 (local address 127.29.106.1:33721)
0811 22:25:28.143795 (+ 1100us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:28.143800 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:28.143845 (+ 45us) server_negotiation.cc:407] Connection header received
0811 22:25:28.143930 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:28.143934 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:28.144013 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:28.144176 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:25:28.144880 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:28.146183 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:28.148258 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:28.149145 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:28.149258 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:28.149840 (+ 582us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:25:28.150504 (+ 664us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:25:28.150661 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:28.150753 (+ 92us) server_negotiation.cc:299] Negotiation successful
0811 22:25:28.150834 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":971,"thread_start_us":771,"threads_started":1}
I20260811 22:25:28.172883 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:28.163529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:47145 (local address 127.29.106.3:42937)
0811 22:25:28.163685 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:28.163689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:28.165508 (+ 1819us) server_negotiation.cc:407] Connection header received
0811 22:25:28.165735 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:28.165738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:28.165804 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:28.165874 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:25:28.166876 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:28.168246 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:28.170788 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:28.171519 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:28.171646 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:28.172188 (+ 542us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:25:28.172286 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:25:28.172349 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:28.172474 (+ 125us) server_negotiation.cc:299] Negotiation successful
0811 22:25:28.172541 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260811 22:25:28.173220 2083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:28.163700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.3:42937 (local address 127.29.106.1:47145)
0811 22:25:28.165405 (+ 1705us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:28.165423 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:28.165608 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:28.165899 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:28.165903 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:28.165912 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 22:25:28.166669 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:28.166681 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:28.168451 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:28.168455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:28.170656 (+ 2201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:28.170670 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:28.171762 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:28.171768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:28.171960 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:28.172017 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 22:25:28.172359 (+ 342us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 22:25:28.172366 (+ 7us) client_negotiation.cc:769] Sending connection context
0811 22:25:28.172419 (+ 53us) client_negotiation.cc:240] Negotiation successful
0811 22:25:28.172532 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1510,"thread_start_us":1000,"threads_started":1}
I20260811 22:25:28.182503 1761 raft_consensus.cc:1261] T 34e2c1c03e394ae99687e823a70da469 P eb362cc342004cdc9d75207b00cbee35 [term 1 FOLLOWER]: Refusing update from remote peer 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:25:28.182574 1468 raft_consensus.cc:1261] T 34e2c1c03e394ae99687e823a70da469 P a669f1e0074e4968bc0853b56071bdc0 [term 1 FOLLOWER]: Refusing update from remote peer 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:25:28.183387 2073 consensus_queue.cc:1035] T 34e2c1c03e394ae99687e823a70da469 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a669f1e0074e4968bc0853b56071bdc0" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:25:28.184005 2046 consensus_queue.cc:1035] T 34e2c1c03e394ae99687e823a70da469 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb362cc342004cdc9d75207b00cbee35" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:25:28.190281 1991 mvcc.cc:205] Tried to move back new op lower bound from 7317451277028503552 to 7317451276641341440. Current Snapshot: MvccSnapshot[applied={T|T < 7317451277028503552}]
I20260811 22:25:28.219893 2076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:28.212428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:39563 (local address 127.29.106.1:33721)
0811 22:25:28.212627 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:28.212634 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:28.213544 (+ 910us) server_negotiation.cc:407] Connection header received
0811 22:25:28.213715 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:28.213721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:28.213798 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:28.213878 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:25:28.214838 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:28.215816 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:28.217963 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:28.218741 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:28.218834 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:28.219248 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:25:28.219379 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:25:28.219443 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:28.219535 (+ 92us) server_negotiation.cc:299] Negotiation successful
0811 22:25:28.219598 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260811 22:25:28.220147 2097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:28.212308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:33721 (local address 127.29.106.3:39563)
0811 22:25:28.213445 (+ 1137us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:28.213461 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:28.213599 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:28.213919 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:28.213924 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:28.213937 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 22:25:28.214674 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:28.214687 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:28.215970 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:28.215974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:28.217826 (+ 1852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:28.217840 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:28.218887 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:28.218892 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:28.219058 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:28.219097 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 22:25:28.219449 (+ 352us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 22:25:28.219456 (+ 7us) client_negotiation.cc:769] Sending connection context
0811 22:25:28.219514 (+ 58us) client_negotiation.cc:240] Negotiation successful
0811 22:25:28.219616 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":973,"thread_start_us":753,"threads_started":1}
W20260811 22:25:28.221361 1991 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260811 22:25:29.839638 872 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:25:29.839963 872 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:25:29.846379 872 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc4020c4fbb7446aadc65b89aeb691ba: Sending DeleteTablet for 3 replicas of tablet 34e2c1c03e394ae99687e823a70da469
I20260811 22:25:29.847303 1132 tablet_service.cc:1514] Processing DeleteTablet for tablet 34e2c1c03e394ae99687e823a70da469 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:25:29 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:55836
I20260811 22:25:29.847473 1427 tablet_service.cc:1514] Processing DeleteTablet for tablet 34e2c1c03e394ae99687e823a70da469 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:25:29 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:39668
I20260811 22:25:29.847790 1706 tablet_service.cc:1514] Processing DeleteTablet for tablet 34e2c1c03e394ae99687e823a70da469 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:25:29 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:37306
I20260811 22:25:29.848263 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 1001
I20260811 22:25:29.858467 2162 tablet_replica.cc:330] T 34e2c1c03e394ae99687e823a70da469 P eb362cc342004cdc9d75207b00cbee35: stopping tablet replica
I20260811 22:25:29.858901 2162 raft_consensus.cc:2229] T 34e2c1c03e394ae99687e823a70da469 P eb362cc342004cdc9d75207b00cbee35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:25:29.859490 2162 raft_consensus.cc:2258] T 34e2c1c03e394ae99687e823a70da469 P eb362cc342004cdc9d75207b00cbee35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:25:29.861177 2162 ts_tablet_manager.cc:1905] T 34e2c1c03e394ae99687e823a70da469 P eb362cc342004cdc9d75207b00cbee35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:25:29.861536 2160 tablet_replica.cc:330] T 34e2c1c03e394ae99687e823a70da469 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: stopping tablet replica
I20260811 22:25:29.861884 2160 raft_consensus.cc:2229] T 34e2c1c03e394ae99687e823a70da469 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [term 1 LEADER]: Raft consensus shutting down.
I20260811 22:25:29.862844 2160 raft_consensus.cc:2258] T 34e2c1c03e394ae99687e823a70da469 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:25:29.863423 2161 tablet_replica.cc:330] T 34e2c1c03e394ae99687e823a70da469 P a669f1e0074e4968bc0853b56071bdc0: stopping tablet replica
I20260811 22:25:29.863783 2161 raft_consensus.cc:2229] T 34e2c1c03e394ae99687e823a70da469 P a669f1e0074e4968bc0853b56071bdc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:25:29.863945 2160 ts_tablet_manager.cc:1905] T 34e2c1c03e394ae99687e823a70da469 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:25:29.864341 2161 raft_consensus.cc:2258] T 34e2c1c03e394ae99687e823a70da469 P a669f1e0074e4968bc0853b56071bdc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:25:29.865765 2161 ts_tablet_manager.cc:1905] T 34e2c1c03e394ae99687e823a70da469 P a669f1e0074e4968bc0853b56071bdc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:25:29.869710 2162 ts_tablet_manager.cc:1918] T 34e2c1c03e394ae99687e823a70da469 P eb362cc342004cdc9d75207b00cbee35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260811 22:25:29.870038 2162 log.cc:1196] T 34e2c1c03e394ae99687e823a70da469 P eb362cc342004cdc9d75207b00cbee35: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-2/wal/wals/34e2c1c03e394ae99687e823a70da469
I20260811 22:25:29.870950 2162 ts_tablet_manager.cc:1939] T 34e2c1c03e394ae99687e823a70da469 P eb362cc342004cdc9d75207b00cbee35: Deleting consensus metadata
I20260811 22:25:29.872546 850 catalog_manager.cc:4841] TS eb362cc342004cdc9d75207b00cbee35 (127.29.106.3:42937): tablet 34e2c1c03e394ae99687e823a70da469 (table test-table [id=5fa4711085074120b51c44fe6a0887a3]) successfully deleted
I20260811 22:25:29.873306 2160 ts_tablet_manager.cc:1918] T 34e2c1c03e394ae99687e823a70da469 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260811 22:25:29.873546 2160 log.cc:1196] T 34e2c1c03e394ae99687e823a70da469 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-0/wal/wals/34e2c1c03e394ae99687e823a70da469
I20260811 22:25:29.874158 2161 ts_tablet_manager.cc:1918] T 34e2c1c03e394ae99687e823a70da469 P a669f1e0074e4968bc0853b56071bdc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260811 22:25:29.874389 2161 log.cc:1196] T 34e2c1c03e394ae99687e823a70da469 P a669f1e0074e4968bc0853b56071bdc0: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786487105757681-30120-0/minicluster-data/ts-1/wal/wals/34e2c1c03e394ae99687e823a70da469
I20260811 22:25:29.874418 2160 ts_tablet_manager.cc:1939] T 34e2c1c03e394ae99687e823a70da469 P 2bb5c6c3715e4f7ebbcca58e9cf0e7f5: Deleting consensus metadata
I20260811 22:25:29.875049 2161 ts_tablet_manager.cc:1939] T 34e2c1c03e394ae99687e823a70da469 P a669f1e0074e4968bc0853b56071bdc0: Deleting consensus metadata
I20260811 22:25:29.875597 850 catalog_manager.cc:4841] TS 2bb5c6c3715e4f7ebbcca58e9cf0e7f5 (127.29.106.1:33721): tablet 34e2c1c03e394ae99687e823a70da469 (table test-table [id=5fa4711085074120b51c44fe6a0887a3]) successfully deleted
I20260811 22:25:29.876024 847 catalog_manager.cc:4841] TS a669f1e0074e4968bc0853b56071bdc0 (127.29.106.2:37479): tablet 34e2c1c03e394ae99687e823a70da469 (table test-table [id=5fa4711085074120b51c44fe6a0887a3]) successfully deleted
Aug 11 22:25:29 dist-test-slave-gjrz krb5kdc[778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487127, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.106.1@KRBTEST.COM
I20260811 22:25:29.887085 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:29.851502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58004 (local address 127.29.106.1:33721)
0811 22:25:29.857328 (+ 5826us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:29.857335 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:29.862588 (+ 5253us) server_negotiation.cc:407] Connection header received
0811 22:25:29.862992 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:29.862998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:29.863097 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:29.863182 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:29.865057 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:29.866683 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:29.869152 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:29.870360 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:29.870462 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:29.878797 (+ 8335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:29.878819 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:29.878842 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:29.878882 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:29.883367 (+ 4485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:29.884151 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:29.884158 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:29.884161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:29.884373 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:29.884796 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:29.884800 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:29.884802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:29.885424 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:29.885939 (+ 515us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:29.886164 (+ 225us) server_negotiation.cc:299] Negotiation successful
0811 22:25:29.886452 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5584,"thread_start_us":5214,"threads_started":1}
W20260811 22:25:29.888015 1270 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58004
W20260811 22:25:29.888859 30120 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 1001: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 22:25:29.889189 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 1001
I20260811 22:25:29.921631 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 1289
Aug 11 22:25:29 dist-test-slave-gjrz krb5kdc[778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487127, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.106.2@KRBTEST.COM
I20260811 22:25:29.936920 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:29.922367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46894 (local address 127.29.106.2:37479)
0811 22:25:29.922849 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:29.922854 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:29.922902 (+ 48us) server_negotiation.cc:407] Connection header received
0811 22:25:29.922960 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:29.922963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:29.923031 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:29.923121 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:29.925139 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:29.925917 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:29.927142 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:29.927580 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:29.927675 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:29.931647 (+ 3972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:29.931662 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:29.931678 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:29.931708 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:29.934504 (+ 2796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:29.935170 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:29.935177 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:29.935179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:29.935440 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:29.935727 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:29.935730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:29.935732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:29.936064 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:29.936280 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:29.936478 (+ 198us) server_negotiation.cc:299] Negotiation successful
0811 22:25:29.936636 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":247,"threads_started":1}
W20260811 22:25:29.937577 1558 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46894
W20260811 22:25:29.938283 30120 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 1289: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 22:25:29.938438 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 1289
I20260811 22:25:29.967767 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 1574
Aug 11 22:25:29 dist-test-slave-gjrz krb5kdc[778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487127, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.106.3@KRBTEST.COM
I20260811 22:25:29.982349 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:29.968534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40000 (local address 127.29.106.3:42937)
0811 22:25:29.969001 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:29.969005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:29.969021 (+ 16us) server_negotiation.cc:407] Connection header received
0811 22:25:29.969072 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:29.969075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:29.969131 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:29.969223 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:29.970513 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:29.971259 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:29.972472 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:29.972890 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:29.973003 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:29.977103 (+ 4100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:29.977118 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:29.977136 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:29.977173 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:29.979916 (+ 2743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:29.980554 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:29.980562 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:29.980565 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:29.980699 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:29.980993 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:29.980996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:29.980998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:29.981344 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:29.981574 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:29.981900 (+ 326us) server_negotiation.cc:299] Negotiation successful
0811 22:25:29.982098 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":223,"threads_started":1}
W20260811 22:25:29.983001 1841 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40000
W20260811 22:25:29.983577 30120 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 1574: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 22:25:29.983731 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 1574
I20260811 22:25:30.014596 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 813
I20260811 22:25:30.015775 944 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:25:30.130661 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 813
2026-08-11T22:25:30Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8581 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:25:30 dist-test-slave-gjrz krb5kdc[2182](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:25:30 dist-test-slave-gjrz krb5kdc[2182](info): set up 2 sockets
Aug 11 22:25:30 dist-test-slave-gjrz krb5kdc[2182](info): commencing operation
krb5kdc: starting...
W20260811 22:25:31.410976 30120 mini_kdc.cc:121] Time spent starting KDC: real 1.229s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260811 22:25:31.550211 30120 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.139s user 0.000s sys 0.007s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260811 22:25:31.787842 30120 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.237s user 0.000s sys 0.006s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260811 22:25:32.001022 30120 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.213s user 0.000s sys 0.007s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/krb5kdc/test-user.keytab.
W20260811 22:25:32.468667 30120 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.467s user 0.000s sys 0.008s
Aug 11 22:25:32 dist-test-slave-gjrz krb5kdc[2182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:25:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:25:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.106.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
I20260811 22:25:32.552883 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:43657
--webserver_interface=127.29.106.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:36819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:43657
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:25:32.871968 2199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:32.872259 2199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:32.872388 2199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:32.879501 2199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:25:32.879599 2199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:32.879673 2199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:32.879755 2199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:25:32.879807 2199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:25:32.889789 2199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:36819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:43657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:43657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.106.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:25:32.891057 2199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:32.892366 2199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:32.903792 2205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:32.903822 2206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:32.903833 2211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:32.904693 2199 server_base.cc:1034] running on GCE node
I20260811 22:25:32.905414 2199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:32.906742 2199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:32.907958 2199 hybrid_clock.cc:648] HybridClock initialized: now 1786487132907905 us; error 27 us; skew 500 ppm
I20260811 22:25:32.908304 2199 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:25:32 dist-test-slave-gjrz krb5kdc[2182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487132, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:32.917152 2199 init.cc:377] Logged in from keytab as kudu/127.29.106.62@KRBTEST.COM (short username kudu)
I20260811 22:25:32.918753 2199 webserver.cc:460] Webserver started at http://127.29.106.62:39311/ using document root <none> and password file <none>
I20260811 22:25:32.919513 2199 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:32.919625 2199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:32.919898 2199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:32.922892 2199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "32ce28273eb1486c939561092a55e141"
format_stamp: "Formatted at 2026-08-11 22:25:32 on dist-test-slave-gjrz"
server_key: "bf07893a1d3a4fcb1fa693b0c8235428"
server_key_iv: "963b00d46b4dafc5c7af5e6a27f56ea6"
server_key_version: "encryptionkey@0"
I20260811 22:25:32.923652 2199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "32ce28273eb1486c939561092a55e141"
format_stamp: "Formatted at 2026-08-11 22:25:32 on dist-test-slave-gjrz"
server_key: "bf07893a1d3a4fcb1fa693b0c8235428"
server_key_iv: "963b00d46b4dafc5c7af5e6a27f56ea6"
server_key_version: "encryptionkey@0"
I20260811 22:25:32.928968 2199 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260811 22:25:32.932379 2220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:32.933669 2199 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260811 22:25:32.933812 2199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "32ce28273eb1486c939561092a55e141"
format_stamp: "Formatted at 2026-08-11 22:25:32 on dist-test-slave-gjrz"
server_key: "bf07893a1d3a4fcb1fa693b0c8235428"
server_key_iv: "963b00d46b4dafc5c7af5e6a27f56ea6"
server_key_version: "encryptionkey@0"
I20260811 22:25:32.933976 2199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:32.976545 2199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:32.980605 2199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:32.980856 2199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:32.998135 2199 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.62:43657
I20260811 22:25:32.998178 2324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.62:43657 every 8 connection(s)
I20260811 22:25:32.999493 2199 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:25:33.003302 2325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:33.009581 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 2199
I20260811 22:25:33.009811 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:25:33.010540 30120 external_mini_cluster.cc:1428] Setting key 952da310371065e1358cb99ae2097e02
I20260811 22:25:33.013584 2325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32ce28273eb1486c939561092a55e141: Bootstrap starting.
I20260811 22:25:33.016871 2325 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32ce28273eb1486c939561092a55e141: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:33.017992 2325 log.cc:824] T 00000000000000000000000000000000 P 32ce28273eb1486c939561092a55e141: Log is configured to *not* fsync() on all Append() calls
I20260811 22:25:33.020828 2325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32ce28273eb1486c939561092a55e141: No bootstrap required, opened a new log
Aug 11 22:25:33 dist-test-slave-gjrz krb5kdc[2182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:25:33.029887 2325 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32ce28273eb1486c939561092a55e141 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32ce28273eb1486c939561092a55e141" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 43657 } }
I20260811 22:25:33.030303 2325 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32ce28273eb1486c939561092a55e141 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:25:33.030416 2325 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32ce28273eb1486c939561092a55e141 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32ce28273eb1486c939561092a55e141, State: Initialized, Role: FOLLOWER
I20260811 22:25:33.031054 2325 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32ce28273eb1486c939561092a55e141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32ce28273eb1486c939561092a55e141" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 43657 } }
I20260811 22:25:33.031371 2325 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32ce28273eb1486c939561092a55e141 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:25:33.031524 2325 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32ce28273eb1486c939561092a55e141 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:25:33.031718 2325 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32ce28273eb1486c939561092a55e141 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:33.033769 2330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:33.012403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40048 (local address 127.29.106.62:43657)
0811 22:25:33.013135 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:33.013146 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:33.013188 (+ 42us) server_negotiation.cc:407] Connection header received
0811 22:25:33.014914 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:33.014940 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:33.015328 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:33.015634 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:33.017436 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:33.018823 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:33.020179 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:33.021178 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:33.021368 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:33.025329 (+ 3961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:33.025343 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:33.025360 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:33.025404 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:33.029415 (+ 4011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:33.030080 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:33.030089 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:33.030097 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:33.030303 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:33.030668 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:33.030672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:33.030675 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:33.031397 (+ 722us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:33.031707 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:33.032047 (+ 340us) server_negotiation.cc:299] Negotiation successful
0811 22:25:33.032482 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":320,"threads_started":1}
I20260811 22:25:33.034891 2325 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32ce28273eb1486c939561092a55e141 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32ce28273eb1486c939561092a55e141" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 43657 } }
I20260811 22:25:33.035471 2325 leader_election.cc:302] T 00000000000000000000000000000000 P 32ce28273eb1486c939561092a55e141 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32ce28273eb1486c939561092a55e141; no voters:
I20260811 22:25:33.036069 2325 leader_election.cc:288] T 00000000000000000000000000000000 P 32ce28273eb1486c939561092a55e141 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:25:33.036216 2335 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32ce28273eb1486c939561092a55e141 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:25:33.037328 2335 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32ce28273eb1486c939561092a55e141 [term 1 LEADER]: Becoming Leader. State: Replica: 32ce28273eb1486c939561092a55e141, State: Running, Role: LEADER
I20260811 22:25:33.037792 2335 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32ce28273eb1486c939561092a55e141 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32ce28273eb1486c939561092a55e141" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 43657 } }
I20260811 22:25:33.038064 2325 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32ce28273eb1486c939561092a55e141 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:25:33.040477 2337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32ce28273eb1486c939561092a55e141 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32ce28273eb1486c939561092a55e141. Latest consensus state: current_term: 1 leader_uuid: "32ce28273eb1486c939561092a55e141" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32ce28273eb1486c939561092a55e141" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 43657 } } }
I20260811 22:25:33.040591 2336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32ce28273eb1486c939561092a55e141 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32ce28273eb1486c939561092a55e141" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32ce28273eb1486c939561092a55e141" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 43657 } } }
I20260811 22:25:33.040853 2337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32ce28273eb1486c939561092a55e141 [sys.catalog]: This master's current role is: LEADER
I20260811 22:25:33.040913 2336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32ce28273eb1486c939561092a55e141 [sys.catalog]: This master's current role is: LEADER
I20260811 22:25:33.043771 2349 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:25:33.048231 2349 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:25:33.056494 2349 catalog_manager.cc:1347] Generated new cluster ID: b00fcc67860d4b5a88c109f92a822f19
I20260811 22:25:33.056595 2349 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:25:33.101503 2349 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:25:33.103428 2349 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:25:33.117281 2349 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32ce28273eb1486c939561092a55e141: Generated new TSK 0
I20260811 22:25:33.118041 2349 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.106.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab.
Entry for principal kudu/127.29.106.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab.
I20260811 22:25:33.172648 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.1:0
--local_ip_for_outbound_sockets=127.29.106.1
--webserver_interface=127.29.106.1
--webserver_port=0
--tserver_master_addrs=127.29.106.62:43657
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab
--principal=kudu/127.29.106.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:36819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:25:33.485877 2370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:33.486140 2370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:33.486270 2370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:33.493170 2370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:33.493278 2370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:33.493424 2370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.1
I20260811 22:25:33.504487 2370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:36819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab
--principal=kudu/127.29.106.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.106.1
--webserver_port=0
--tserver_master_addrs=127.29.106.62:43657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.1
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:25:33.505607 2370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:33.506804 2370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:33.518949 2386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:33.518975 2387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:33.518913 2390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:33.519753 2370 server_base.cc:1034] running on GCE node
I20260811 22:25:33.520289 2370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:33.521116 2370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:33.522300 2370 hybrid_clock.cc:648] HybridClock initialized: now 1786487133522258 us; error 33 us; skew 500 ppm
I20260811 22:25:33.522576 2370 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:25:33 dist-test-slave-gjrz krb5kdc[2182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487133, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:33.525458 2370 init.cc:377] Logged in from keytab as kudu/127.29.106.1@KRBTEST.COM (short username kudu)
I20260811 22:25:33.526715 2370 webserver.cc:460] Webserver started at http://127.29.106.1:36247/ using document root <none> and password file <none>
I20260811 22:25:33.527346 2370 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:33.527448 2370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:33.527697 2370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:33.530520 2370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-0/data/instance:
uuid: "0685d7c0c3084c868d36ad99c3d25bdb"
format_stamp: "Formatted at 2026-08-11 22:25:33 on dist-test-slave-gjrz"
server_key: "2e78f63932c642748d3da9e8312977af"
server_key_iv: "b56b26b5d80005ab7754f7be47ea119d"
server_key_version: "encryptionkey@0"
I20260811 22:25:33.531236 2370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance:
uuid: "0685d7c0c3084c868d36ad99c3d25bdb"
format_stamp: "Formatted at 2026-08-11 22:25:33 on dist-test-slave-gjrz"
server_key: "2e78f63932c642748d3da9e8312977af"
server_key_iv: "b56b26b5d80005ab7754f7be47ea119d"
server_key_version: "encryptionkey@0"
I20260811 22:25:33.535985 2370 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260811 22:25:33.539353 2401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:33.540700 2370 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260811 22:25:33.540848 2370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-0/wal
uuid: "0685d7c0c3084c868d36ad99c3d25bdb"
format_stamp: "Formatted at 2026-08-11 22:25:33 on dist-test-slave-gjrz"
server_key: "2e78f63932c642748d3da9e8312977af"
server_key_iv: "b56b26b5d80005ab7754f7be47ea119d"
server_key_version: "encryptionkey@0"
I20260811 22:25:33.541031 2370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:33.567778 2370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:33.571491 2370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:33.571740 2370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:33.572605 2370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:25:33.573872 2370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:25:33.573943 2370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:33.574061 2370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:25:33.574115 2370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:33.604174 2370 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.1:42641
I20260811 22:25:33.604238 2614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.1:42641 every 8 connection(s)
I20260811 22:25:33.605351 2370 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
I20260811 22:25:33.610015 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 2370
I20260811 22:25:33.610212 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance
I20260811 22:25:33.610781 30120 external_mini_cluster.cc:1428] Setting key 0452dc1318ec685ea71783c21b035d85
Aug 11 22:25:33 dist-test-slave-gjrz krb5kdc[2182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487133, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.1@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:25:33.624655 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:33.607696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:34141 (local address 127.29.106.62:43657)
0811 22:25:33.608148 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:33.608153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:33.609022 (+ 869us) server_negotiation.cc:407] Connection header received
0811 22:25:33.610893 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:33.610899 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:33.610989 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:33.611097 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:33.613006 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:33.613753 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:33.615318 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:33.615736 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:33.615849 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:33.619319 (+ 3470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:33.619326 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:33.619330 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:33.619360 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:33.621711 (+ 2351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:33.622406 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:33.622410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:33.622413 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:33.622533 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:33.622953 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:33.622958 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:33.622960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:33.623222 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:33.623403 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:33.624032 (+ 629us) server_negotiation.cc:299] Negotiation successful
0811 22:25:33.624256 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":217,"threads_started":1}
I20260811 22:25:33.625521 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:33.608208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:43657 (local address 127.29.106.1:34141)
0811 22:25:33.608870 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:33.608915 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:33.610618 (+ 1703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:33.611384 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:33.611395 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:33.612061 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:33.612818 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:33.612844 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:33.613917 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:33.613923 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:33.615150 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:33.615167 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:33.615962 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:33.615970 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:33.616214 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:33.616809 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:33.616837 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:33.619052 (+ 2215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:33.621842 (+ 2790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:33.621851 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:33.621866 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:33.622286 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:33.622636 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:33.622649 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:33.622653 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:33.622833 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:33.623364 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:33.623371 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:33.623685 (+ 314us) client_negotiation.cc:769] Sending connection context
0811 22:25:33.623931 (+ 246us) client_negotiation.cc:240] Negotiation successful
0811 22:25:33.624408 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":253,"threads_started":1}
I20260811 22:25:33.627020 2616 heartbeater.cc:347] Connected to a master server at 127.29.106.62:43657
I20260811 22:25:33.627322 2616 heartbeater.cc:464] Registering TS with master...
I20260811 22:25:33.627991 2616 heartbeater.cc:511] Master 127.29.106.62:43657 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.29.106.2@KRBTEST.COM; defaulting to no policy
I20260811 22:25:33.630036 2248 ts_manager.cc:194] Registered new tserver with Master: 0685d7c0c3084c868d36ad99c3d25bdb (127.29.106.1:42641)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.2@KRBTEST.COM" created.
I20260811 22:25:33.631494 2248 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.1@KRBTEST.COM'} at 127.29.106.1:34141
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab.
Entry for principal kudu/127.29.106.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab.
I20260811 22:25:33.652340 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.2:0
--local_ip_for_outbound_sockets=127.29.106.2
--webserver_interface=127.29.106.2
--webserver_port=0
--tserver_master_addrs=127.29.106.62:43657
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab
--principal=kudu/127.29.106.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:36819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:25:33.960093 2626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:33.960333 2626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:33.960445 2626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:33.967407 2626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:33.967504 2626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:33.967651 2626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.2
I20260811 22:25:33.976802 2626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:36819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab
--principal=kudu/127.29.106.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.106.2
--webserver_port=0
--tserver_master_addrs=127.29.106.62:43657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.2
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:25:33.977953 2626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:33.979169 2626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:33.991330 2638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:33.991400 2640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:33.991369 2637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:33.992089 2626 server_base.cc:1034] running on GCE node
I20260811 22:25:33.992610 2626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:33.993434 2626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:33.994647 2626 hybrid_clock.cc:648] HybridClock initialized: now 1786487133994596 us; error 35 us; skew 500 ppm
I20260811 22:25:33.994948 2626 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:25:33 dist-test-slave-gjrz krb5kdc[2182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487133, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:33.997992 2626 init.cc:377] Logged in from keytab as kudu/127.29.106.2@KRBTEST.COM (short username kudu)
I20260811 22:25:33.999369 2626 webserver.cc:460] Webserver started at http://127.29.106.2:40871/ using document root <none> and password file <none>
I20260811 22:25:33.999989 2626 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:34.000105 2626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:34.000396 2626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:34.003144 2626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-1/data/instance:
uuid: "d6a50b949eb8418aaa3b4cbf3eda429f"
format_stamp: "Formatted at 2026-08-11 22:25:33 on dist-test-slave-gjrz"
server_key: "c8cc07548f0de1c4a1327c512f85b435"
server_key_iv: "d31de45929629a2e9c1b8ef0cb2233db"
server_key_version: "encryptionkey@0"
I20260811 22:25:34.003937 2626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance:
uuid: "d6a50b949eb8418aaa3b4cbf3eda429f"
format_stamp: "Formatted at 2026-08-11 22:25:33 on dist-test-slave-gjrz"
server_key: "c8cc07548f0de1c4a1327c512f85b435"
server_key_iv: "d31de45929629a2e9c1b8ef0cb2233db"
server_key_version: "encryptionkey@0"
I20260811 22:25:34.008852 2626 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260811 22:25:34.012146 2651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:34.013371 2626 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 22:25:34.013521 2626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-1/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-1/wal
uuid: "d6a50b949eb8418aaa3b4cbf3eda429f"
format_stamp: "Formatted at 2026-08-11 22:25:33 on dist-test-slave-gjrz"
server_key: "c8cc07548f0de1c4a1327c512f85b435"
server_key_iv: "d31de45929629a2e9c1b8ef0cb2233db"
server_key_version: "encryptionkey@0"
I20260811 22:25:34.013696 2626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:34.048489 2626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:34.052313 2626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:34.052585 2626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:34.053510 2626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:25:34.054778 2626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:25:34.054859 2626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:34.054975 2626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:25:34.055024 2626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:34.084980 2626 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.2:46001
I20260811 22:25:34.085039 2872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.2:46001 every 8 connection(s)
I20260811 22:25:34.086179 2626 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
I20260811 22:25:34.088811 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 2626
I20260811 22:25:34.089004 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance
I20260811 22:25:34.089561 30120 external_mini_cluster.cc:1428] Setting key e2e62d7ea527cbee8b18567b05af9e1f
Aug 11 22:25:34 dist-test-slave-gjrz krb5kdc[2182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487133, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.2@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:25:34.105487 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:34.088645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:59987 (local address 127.29.106.62:43657)
0811 22:25:34.088815 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:34.088820 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:34.090144 (+ 1324us) server_negotiation.cc:407] Connection header received
0811 22:25:34.091972 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:34.091977 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:34.092061 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:34.092147 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:34.093565 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:34.094302 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:34.095853 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:34.096257 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:34.096365 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:34.099866 (+ 3501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:34.099870 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:34.099874 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:34.099904 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:34.102495 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:34.103179 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:34.103182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:34.103185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:34.103350 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:34.103771 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:34.103776 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:34.103778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:34.104068 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:34.104210 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:34.104921 (+ 711us) server_negotiation.cc:299] Negotiation successful
0811 22:25:34.105165 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 22:25:34.106329 2879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:34.089227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:43657 (local address 127.29.106.2:59987)
0811 22:25:34.089985 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:34.090022 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:34.091708 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:34.092377 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:34.092384 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:34.092858 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:34.093391 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:34.093412 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:34.094446 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:34.094450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:34.095710 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:34.095723 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:34.096465 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:34.096474 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:34.096696 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:34.097304 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:34.097325 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:34.099663 (+ 2338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:34.102627 (+ 2964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:34.102636 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:34.102651 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:34.103059 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:34.103451 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:34.103456 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:34.103460 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:34.103658 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:34.104192 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:34.104199 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:34.104575 (+ 376us) client_negotiation.cc:769] Sending connection context
0811 22:25:34.104803 (+ 228us) client_negotiation.cc:240] Negotiation successful
0811 22:25:34.105279 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":317,"threads_started":1}
I20260811 22:25:34.107699 2873 heartbeater.cc:347] Connected to a master server at 127.29.106.62:43657
I20260811 22:25:34.108045 2873 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.29.106.3@KRBTEST.COM; defaulting to no policy
I20260811 22:25:34.108850 2873 heartbeater.cc:511] Master 127.29.106.62:43657 requested a full tablet report, sending...
I20260811 22:25:34.110333 2248 ts_manager.cc:194] Registered new tserver with Master: d6a50b949eb8418aaa3b4cbf3eda429f (127.29.106.2:46001)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.3@KRBTEST.COM" created.
I20260811 22:25:34.111212 2248 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.2@KRBTEST.COM'} at 127.29.106.2:59987
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab.
Entry for principal kudu/127.29.106.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab.
I20260811 22:25:34.132437 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.3:0
--local_ip_for_outbound_sockets=127.29.106.3
--webserver_interface=127.29.106.3
--webserver_port=0
--tserver_master_addrs=127.29.106.62:43657
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab
--principal=kudu/127.29.106.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:36819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:25:34.441133 2884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:34.441381 2884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:34.441520 2884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:34.448659 2884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:34.448774 2884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:34.448926 2884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.3
I20260811 22:25:34.458560 2884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:36819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab
--principal=kudu/127.29.106.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.106.3
--webserver_port=0
--tserver_master_addrs=127.29.106.62:43657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.3
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:25:34.459808 2884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:34.461072 2884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:34.473274 2891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:34.473243 2893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:34.473315 2895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:34.474315 2884 server_base.cc:1034] running on GCE node
I20260811 22:25:34.474857 2884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:34.475746 2884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:34.476939 2884 hybrid_clock.cc:648] HybridClock initialized: now 1786487134476897 us; error 37 us; skew 500 ppm
I20260811 22:25:34.477221 2884 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:25:34 dist-test-slave-gjrz krb5kdc[2182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487134, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:34.480073 2884 init.cc:377] Logged in from keytab as kudu/127.29.106.3@KRBTEST.COM (short username kudu)
I20260811 22:25:34.481374 2884 webserver.cc:460] Webserver started at http://127.29.106.3:33041/ using document root <none> and password file <none>
I20260811 22:25:34.482008 2884 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:34.482123 2884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:34.482403 2884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:34.485126 2884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-2/data/instance:
uuid: "8df295bf1cf549109568955b86d07fd1"
format_stamp: "Formatted at 2026-08-11 22:25:34 on dist-test-slave-gjrz"
server_key: "b1d8b1b5fb47d90c6a7260ec40a7480c"
server_key_iv: "f22db6b5854f52270caf3f4b5fb8a346"
server_key_version: "encryptionkey@0"
I20260811 22:25:34.485859 2884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance:
uuid: "8df295bf1cf549109568955b86d07fd1"
format_stamp: "Formatted at 2026-08-11 22:25:34 on dist-test-slave-gjrz"
server_key: "b1d8b1b5fb47d90c6a7260ec40a7480c"
server_key_iv: "f22db6b5854f52270caf3f4b5fb8a346"
server_key_version: "encryptionkey@0"
I20260811 22:25:34.490711 2884 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260811 22:25:34.494059 2910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:34.495298 2884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260811 22:25:34.495446 2884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-2/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-2/wal
uuid: "8df295bf1cf549109568955b86d07fd1"
format_stamp: "Formatted at 2026-08-11 22:25:34 on dist-test-slave-gjrz"
server_key: "b1d8b1b5fb47d90c6a7260ec40a7480c"
server_key_iv: "f22db6b5854f52270caf3f4b5fb8a346"
server_key_version: "encryptionkey@0"
I20260811 22:25:34.495626 2884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:34.520766 2884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:34.524770 2884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:34.525038 2884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:34.525883 2884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:25:34.527124 2884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:25:34.527201 2884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:34.527354 2884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:25:34.527410 2884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:34.557397 2884 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.3:45691
I20260811 22:25:34.557441 3128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.3:45691 every 8 connection(s)
I20260811 22:25:34.558547 2884 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
I20260811 22:25:34.568351 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 2884
I20260811 22:25:34.568542 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance
I20260811 22:25:34.568935 30120 external_mini_cluster.cc:1428] Setting key 9bf29b9fd16df32640584ac66a8d6226
Aug 11 22:25:34 dist-test-slave-gjrz krb5kdc[2182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487134, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.3@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:25:34.576951 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:34.560845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:45349 (local address 127.29.106.62:43657)
0811 22:25:34.561013 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:34.561017 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:34.562143 (+ 1126us) server_negotiation.cc:407] Connection header received
0811 22:25:34.563994 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:34.564000 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:34.564089 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:34.564176 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:34.565583 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:34.566330 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:34.567867 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:34.568311 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:34.568420 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:34.571954 (+ 3534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:34.571959 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:34.571963 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:34.571994 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:34.574236 (+ 2242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:34.574947 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:34.574952 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:34.574954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:34.575082 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:34.575535 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:34.575539 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:34.575541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:34.575808 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:34.575929 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:34.576547 (+ 618us) server_negotiation.cc:299] Negotiation successful
0811 22:25:34.576719 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260811 22:25:34.577855 3131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:34.561284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:43657 (local address 127.29.106.3:45349)
0811 22:25:34.561983 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:34.562025 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:34.563729 (+ 1704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:34.564407 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:34.564415 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:34.564891 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:34.565415 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:34.565432 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:34.566475 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:34.566479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:34.567724 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:34.567742 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:34.568534 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:34.568542 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:34.568789 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:34.569417 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:34.569438 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:34.571751 (+ 2313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:34.574374 (+ 2623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:34.574381 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:34.574395 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:34.574803 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:34.575179 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:34.575182 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:34.575185 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:34.575407 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:34.575936 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:34.575965 (+ 29us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:34.576248 (+ 283us) client_negotiation.cc:769] Sending connection context
0811 22:25:34.576460 (+ 212us) client_negotiation.cc:240] Negotiation successful
0811 22:25:34.576801 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":278,"threads_started":1}
I20260811 22:25:34.579216 3129 heartbeater.cc:347] Connected to a master server at 127.29.106.62:43657
I20260811 22:25:34.579493 3129 heartbeater.cc:464] Registering TS with master...
I20260811 22:25:34.580123 3129 heartbeater.cc:511] Master 127.29.106.62:43657 requested a full tablet report, sending...
I20260811 22:25:34.581429 2248 ts_manager.cc:194] Registered new tserver with Master: 8df295bf1cf549109568955b86d07fd1 (127.29.106.3:45691)
I20260811 22:25:34.582228 2248 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.3@KRBTEST.COM'} at 127.29.106.3:45349
I20260811 22:25:34.583995 30120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260811 22:25:34.617005 2618 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:25:34.614648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40060 (local address 127.29.106.62:43657)
0811 22:25:34.614862 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:34.614869 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:34.615225 (+ 356us) server_negotiation.cc:407] Connection header received
0811 22:25:34.615428 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:34.615432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:34.615501 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:34.615584 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:34.616752 (+ 1168us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40060: BlockingRecv error: recv got EOF from 127.0.0.1:40060 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260811 22:25:34.620297 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 2370
W20260811 22:25:34.622807 30120 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 2370: Not authorized: Client connection negotiation failed: client connection to 127.29.106.1:42641: server requires authentication, but client does not have Kerberos credentials available
I20260811 22:25:34.622994 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 2370
I20260811 22:25:34.647403 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 2626
W20260811 22:25:34.649619 30120 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 2626: Not authorized: Client connection negotiation failed: client connection to 127.29.106.2:46001: server requires authentication, but client does not have Kerberos credentials available
I20260811 22:25:34.649749 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 2626
I20260811 22:25:34.672451 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 2884
W20260811 22:25:34.675074 30120 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 2884: Not authorized: Client connection negotiation failed: client connection to 127.29.106.3:45691: server requires authentication, but client does not have Kerberos credentials available
I20260811 22:25:34.675194 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 2884
I20260811 22:25:34.697059 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 2199
I20260811 22:25:34.698181 2315 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:25:34.800768 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 2199
2026-08-11T22:25:34Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (4667 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786487105757681-30120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:25:34 dist-test-slave-gjrz krb5kdc[3162](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:25:34 dist-test-slave-gjrz krb5kdc[3162](info): set up 2 sockets
Aug 11 22:25:34 dist-test-slave-gjrz krb5kdc[3162](info): commencing operation
krb5kdc: starting...
W20260811 22:25:36.037797 30120 mini_kdc.cc:121] Time spent starting KDC: real 1.189s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Aug 11 22:25:36 dist-test-slave-gjrz krb5kdc[3162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:25:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:25:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.106.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
I20260811 22:25:36.230041 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:43701
--webserver_interface=127.29.106.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:38119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:43701
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:25:36.544468 3182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:36.544729 3182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:36.544824 3182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:36.551759 3182 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:25:36.551847 3182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:36.551898 3182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:36.551960 3182 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:25:36.552007 3182 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:25:36.561591 3182 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:38119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:43701
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:43701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.106.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:25:36.562721 3182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:36.563959 3182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:36.575076 3189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:36.575228 3193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:36.575248 3190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:36.576004 3182 server_base.cc:1034] running on GCE node
I20260811 22:25:36.576658 3182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:36.577922 3182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:36.579126 3182 hybrid_clock.cc:648] HybridClock initialized: now 1786487136579070 us; error 44 us; skew 500 ppm
I20260811 22:25:36.579479 3182 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:25:36 dist-test-slave-gjrz krb5kdc[3162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487136, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:36.582363 3182 init.cc:377] Logged in from keytab as kudu/127.29.106.62@KRBTEST.COM (short username kudu)
I20260811 22:25:36.583774 3182 webserver.cc:460] Webserver started at http://127.29.106.62:37643/ using document root <none> and password file <none>
I20260811 22:25:36.584393 3182 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:36.584511 3182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:36.584794 3182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:36.587666 3182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "9cf1dd41089d4b7992b053fb79dd8038"
format_stamp: "Formatted at 2026-08-11 22:25:36 on dist-test-slave-gjrz"
server_key: "5560873fac03c6075564499fe2dbd2b5"
server_key_iv: "37147d5549a7b83abbe5cb0f40937b23"
server_key_version: "encryptionkey@0"
I20260811 22:25:36.588419 3182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "9cf1dd41089d4b7992b053fb79dd8038"
format_stamp: "Formatted at 2026-08-11 22:25:36 on dist-test-slave-gjrz"
server_key: "5560873fac03c6075564499fe2dbd2b5"
server_key_iv: "37147d5549a7b83abbe5cb0f40937b23"
server_key_version: "encryptionkey@0"
I20260811 22:25:36.593272 3182 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 22:25:36.596448 3203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:36.597626 3182 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:25:36.597774 3182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "9cf1dd41089d4b7992b053fb79dd8038"
format_stamp: "Formatted at 2026-08-11 22:25:36 on dist-test-slave-gjrz"
server_key: "5560873fac03c6075564499fe2dbd2b5"
server_key_iv: "37147d5549a7b83abbe5cb0f40937b23"
server_key_version: "encryptionkey@0"
I20260811 22:25:36.597980 3182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:36.622990 3182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:36.626500 3182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:36.626745 3182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:36.643790 3182 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.62:43701
I20260811 22:25:36.643822 3305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.62:43701 every 8 connection(s)
I20260811 22:25:36.645041 3182 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:25:36.646135 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 3182
I20260811 22:25:36.646385 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:25:36.647184 30120 external_mini_cluster.cc:1428] Setting key 7f4aad158629ec2d7f4e63b5c8f1f89f
I20260811 22:25:36.649503 3306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 11 22:25:36 dist-test-slave-gjrz krb5kdc[3162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:25:36.661830 3306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cf1dd41089d4b7992b053fb79dd8038: Bootstrap starting.
I20260811 22:25:36.664945 3306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9cf1dd41089d4b7992b053fb79dd8038: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:36.665907 3306 log.cc:824] T 00000000000000000000000000000000 P 9cf1dd41089d4b7992b053fb79dd8038: Log is configured to *not* fsync() on all Append() calls
I20260811 22:25:36.668521 3306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cf1dd41089d4b7992b053fb79dd8038: No bootstrap required, opened a new log
I20260811 22:25:36.668696 3312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:36.648990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42270 (local address 127.29.106.62:43701)
0811 22:25:36.649806 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:36.649817 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:36.649856 (+ 39us) server_negotiation.cc:407] Connection header received
0811 22:25:36.651511 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:36.651531 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:36.651891 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:36.652194 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:36.653981 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:36.655314 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:36.657024 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:36.657639 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:36.657755 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:36.662082 (+ 4327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:36.662091 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:36.662103 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:36.662136 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:36.664970 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:36.665598 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:36.665604 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:36.665610 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:36.665772 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:36.666121 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:36.666126 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:36.666129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:36.666663 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:36.666951 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:36.667430 (+ 479us) server_negotiation.cc:299] Negotiation successful
0811 22:25:36.667798 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":337,"threads_started":1}
I20260811 22:25:36.672122 30120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:25:36.676193 3306 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cf1dd41089d4b7992b053fb79dd8038 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cf1dd41089d4b7992b053fb79dd8038" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 43701 } }
I20260811 22:25:36.676507 3306 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9cf1dd41089d4b7992b053fb79dd8038 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:25:36.676599 3306 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cf1dd41089d4b7992b053fb79dd8038 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cf1dd41089d4b7992b053fb79dd8038, State: Initialized, Role: FOLLOWER
I20260811 22:25:36.677124 3306 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cf1dd41089d4b7992b053fb79dd8038 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cf1dd41089d4b7992b053fb79dd8038" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 43701 } }
I20260811 22:25:36.677352 3306 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9cf1dd41089d4b7992b053fb79dd8038 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:25:36.677472 3306 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cf1dd41089d4b7992b053fb79dd8038 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:25:36.677624 3306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cf1dd41089d4b7992b053fb79dd8038 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:36.679672 3306 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cf1dd41089d4b7992b053fb79dd8038 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cf1dd41089d4b7992b053fb79dd8038" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 43701 } }
I20260811 22:25:36.680115 3306 leader_election.cc:302] T 00000000000000000000000000000000 P 9cf1dd41089d4b7992b053fb79dd8038 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cf1dd41089d4b7992b053fb79dd8038; no voters:
I20260811 22:25:36.680683 3306 leader_election.cc:288] T 00000000000000000000000000000000 P 9cf1dd41089d4b7992b053fb79dd8038 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:25:36.680778 3316 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cf1dd41089d4b7992b053fb79dd8038 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:25:36.681891 3316 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9cf1dd41089d4b7992b053fb79dd8038 [term 1 LEADER]: Becoming Leader. State: Replica: 9cf1dd41089d4b7992b053fb79dd8038, State: Running, Role: LEADER
I20260811 22:25:36.682462 3316 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cf1dd41089d4b7992b053fb79dd8038 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cf1dd41089d4b7992b053fb79dd8038" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 43701 } }
I20260811 22:25:36.682860 3306 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cf1dd41089d4b7992b053fb79dd8038 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:25:36.685236 3317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cf1dd41089d4b7992b053fb79dd8038 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9cf1dd41089d4b7992b053fb79dd8038" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cf1dd41089d4b7992b053fb79dd8038" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 43701 } } }
I20260811 22:25:36.685607 3317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cf1dd41089d4b7992b053fb79dd8038 [sys.catalog]: This master's current role is: LEADER
I20260811 22:25:36.691079 3318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cf1dd41089d4b7992b053fb79dd8038 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cf1dd41089d4b7992b053fb79dd8038. Latest consensus state: current_term: 1 leader_uuid: "9cf1dd41089d4b7992b053fb79dd8038" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cf1dd41089d4b7992b053fb79dd8038" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 43701 } } }
I20260811 22:25:36.691849 3318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cf1dd41089d4b7992b053fb79dd8038 [sys.catalog]: This master's current role is: LEADER
I20260811 22:25:36.692173 3325 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:25:36.699795 3325 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:25:36.718043 3325 catalog_manager.cc:1347] Generated new cluster ID: d70a837d749e49df86ae4846d8285939
I20260811 22:25:36.718173 3325 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:25:36.729616 3325 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:25:36.731253 3325 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:25:36.736351 3312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:36.724998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42274 (local address 127.29.106.62:43701)
0811 22:25:36.725165 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:36.725171 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:36.727853 (+ 2682us) server_negotiation.cc:407] Connection header received
0811 22:25:36.728083 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:36.728088 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:36.728174 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:36.728289 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:36.729691 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:36.730797 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:36.732652 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:36.733309 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:36.733434 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:36.734608 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:36.734615 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:36.734623 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:25:36.735557 (+ 934us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:36.735612 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:25:36.735645 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:36.735740 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:36.735893 (+ 153us) server_negotiation.cc:299] Negotiation successful
0811 22:25:36.736013 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 22:25:36.746783 3325 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9cf1dd41089d4b7992b053fb79dd8038: Generated new TSK 0
I20260811 22:25:36.747753 3325 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:25:36.760854 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 3182
I20260811 22:25:36.762099 3300 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:25:36.851955 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 3182
2026-08-11T22:25:36Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2045 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:25:36 dist-test-slave-gjrz krb5kdc[3363](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:25:36 dist-test-slave-gjrz krb5kdc[3363](info): set up 2 sockets
Aug 11 22:25:36 dist-test-slave-gjrz krb5kdc[3363](info): commencing operation
krb5kdc: starting...
W20260811 22:25:38.082490 30120 mini_kdc.cc:121] Time spent starting KDC: real 1.188s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Aug 11 22:25:38 dist-test-slave-gjrz krb5kdc[3363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:25:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:25:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.106.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
I20260811 22:25:38.268067 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:45339
--webserver_interface=127.29.106.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:35393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:45339
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:25:38.582484 3383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:38.582753 3383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:38.582854 3383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:38.590037 3383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:25:38.590138 3383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:38.590184 3383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:38.590241 3383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:25:38.590284 3383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:25:38.599852 3383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:35393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:45339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:45339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.106.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:25:38.601066 3383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:38.602293 3383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:38.613349 3390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:38.613389 3393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:38.613426 3395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:38.614360 3383 server_base.cc:1034] running on GCE node
I20260811 22:25:38.615093 3383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:38.616466 3383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:38.617686 3383 hybrid_clock.cc:648] HybridClock initialized: now 1786487138617619 us; error 49 us; skew 500 ppm
I20260811 22:25:38.618011 3383 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:25:38 dist-test-slave-gjrz krb5kdc[3363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487138, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:38.625798 3383 init.cc:377] Logged in from keytab as kudu/127.29.106.62@KRBTEST.COM (short username kudu)
I20260811 22:25:38.627215 3383 webserver.cc:460] Webserver started at http://127.29.106.62:45297/ using document root <none> and password file <none>
I20260811 22:25:38.627897 3383 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:38.628012 3383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:38.628314 3383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:38.631187 3383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "f2c7f8df5f4d4f90b0f195b0e3c59d5f"
format_stamp: "Formatted at 2026-08-11 22:25:38 on dist-test-slave-gjrz"
server_key: "05247d9dd412c27ed4e86cb56ab30e21"
server_key_iv: "ab17ef0e1fc37a8a835e18f5514d4ddb"
server_key_version: "encryptionkey@0"
I20260811 22:25:38.631989 3383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "f2c7f8df5f4d4f90b0f195b0e3c59d5f"
format_stamp: "Formatted at 2026-08-11 22:25:38 on dist-test-slave-gjrz"
server_key: "05247d9dd412c27ed4e86cb56ab30e21"
server_key_iv: "ab17ef0e1fc37a8a835e18f5514d4ddb"
server_key_version: "encryptionkey@0"
I20260811 22:25:38.636929 3383 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260811 22:25:38.640151 3405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:38.641396 3383 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260811 22:25:38.641574 3383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "f2c7f8df5f4d4f90b0f195b0e3c59d5f"
format_stamp: "Formatted at 2026-08-11 22:25:38 on dist-test-slave-gjrz"
server_key: "05247d9dd412c27ed4e86cb56ab30e21"
server_key_iv: "ab17ef0e1fc37a8a835e18f5514d4ddb"
server_key_version: "encryptionkey@0"
I20260811 22:25:38.641750 3383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:38.668787 3383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:38.672595 3383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:38.672869 3383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:38.690112 3383 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.62:45339
I20260811 22:25:38.690135 3519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.62:45339 every 8 connection(s)
I20260811 22:25:38.691416 3383 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:25:38.693080 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 3383
I20260811 22:25:38.693292 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:25:38.694094 30120 external_mini_cluster.cc:1428] Setting key 2f0e57b7fe38e854fec2469f4099240b
I20260811 22:25:38.695773 3520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 11 22:25:38 dist-test-slave-gjrz krb5kdc[3363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:25:38.709301 3520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2c7f8df5f4d4f90b0f195b0e3c59d5f: Bootstrap starting.
I20260811 22:25:38.712356 3520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2c7f8df5f4d4f90b0f195b0e3c59d5f: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:38.713577 3520 log.cc:824] T 00000000000000000000000000000000 P f2c7f8df5f4d4f90b0f195b0e3c59d5f: Log is configured to *not* fsync() on all Append() calls
I20260811 22:25:38.716619 3525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:38.696060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43142 (local address 127.29.106.62:45339)
0811 22:25:38.696812 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:38.696823 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:38.696862 (+ 39us) server_negotiation.cc:407] Connection header received
0811 22:25:38.698407 (+ 1545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:38.698426 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:38.698791 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:38.699106 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:38.700871 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:38.702197 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:38.704015 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:38.704679 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:38.704818 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:38.709481 (+ 4663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:38.709489 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:38.709503 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:38.709544 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:38.712324 (+ 2780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:38.713089 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:38.713095 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:38.713102 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:38.713269 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:38.713681 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:38.713686 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:38.713689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:38.714287 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:38.714606 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:38.714913 (+ 307us) server_negotiation.cc:299] Negotiation successful
0811 22:25:38.715422 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":346,"threads_started":1}
I20260811 22:25:38.716787 3520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2c7f8df5f4d4f90b0f195b0e3c59d5f: No bootstrap required, opened a new log
I20260811 22:25:38.723881 3520 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2c7f8df5f4d4f90b0f195b0e3c59d5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2c7f8df5f4d4f90b0f195b0e3c59d5f" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 45339 } }
I20260811 22:25:38.724247 3520 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2c7f8df5f4d4f90b0f195b0e3c59d5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:25:38.724347 3520 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2c7f8df5f4d4f90b0f195b0e3c59d5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2c7f8df5f4d4f90b0f195b0e3c59d5f, State: Initialized, Role: FOLLOWER
I20260811 22:25:38.724895 3520 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2c7f8df5f4d4f90b0f195b0e3c59d5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2c7f8df5f4d4f90b0f195b0e3c59d5f" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 45339 } }
I20260811 22:25:38.725122 3520 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2c7f8df5f4d4f90b0f195b0e3c59d5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:25:38.725242 3520 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2c7f8df5f4d4f90b0f195b0e3c59d5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:25:38.725394 3520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2c7f8df5f4d4f90b0f195b0e3c59d5f [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:38.727473 3520 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2c7f8df5f4d4f90b0f195b0e3c59d5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2c7f8df5f4d4f90b0f195b0e3c59d5f" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 45339 } }
I20260811 22:25:38.727929 3520 leader_election.cc:302] T 00000000000000000000000000000000 P f2c7f8df5f4d4f90b0f195b0e3c59d5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2c7f8df5f4d4f90b0f195b0e3c59d5f; no voters:
I20260811 22:25:38.728497 3520 leader_election.cc:288] T 00000000000000000000000000000000 P f2c7f8df5f4d4f90b0f195b0e3c59d5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:25:38.728631 3529 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2c7f8df5f4d4f90b0f195b0e3c59d5f [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:25:38.729751 3529 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2c7f8df5f4d4f90b0f195b0e3c59d5f [term 1 LEADER]: Becoming Leader. State: Replica: f2c7f8df5f4d4f90b0f195b0e3c59d5f, State: Running, Role: LEADER
I20260811 22:25:38.730221 3529 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2c7f8df5f4d4f90b0f195b0e3c59d5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2c7f8df5f4d4f90b0f195b0e3c59d5f" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 45339 } }
I20260811 22:25:38.730571 3520 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2c7f8df5f4d4f90b0f195b0e3c59d5f [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:25:38.732898 3532 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2c7f8df5f4d4f90b0f195b0e3c59d5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2c7f8df5f4d4f90b0f195b0e3c59d5f. Latest consensus state: current_term: 1 leader_uuid: "f2c7f8df5f4d4f90b0f195b0e3c59d5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2c7f8df5f4d4f90b0f195b0e3c59d5f" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 45339 } } }
I20260811 22:25:38.733376 3532 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2c7f8df5f4d4f90b0f195b0e3c59d5f [sys.catalog]: This master's current role is: LEADER
I20260811 22:25:38.732918 3531 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2c7f8df5f4d4f90b0f195b0e3c59d5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2c7f8df5f4d4f90b0f195b0e3c59d5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2c7f8df5f4d4f90b0f195b0e3c59d5f" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 45339 } } }
I20260811 22:25:38.734093 3531 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2c7f8df5f4d4f90b0f195b0e3c59d5f [sys.catalog]: This master's current role is: LEADER
W20260811 22:25:38.741093 3561 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f2c7f8df5f4d4f90b0f195b0e3c59d5f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 22:25:38.741194 3561 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 22:25:38.741318 3550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:25:38.742874 3550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:25:38.750387 3550 catalog_manager.cc:1347] Generated new cluster ID: 7648ae451c314b05b9c7ccf967efa046
I20260811 22:25:38.750491 3550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:25:38.774350 3550 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:25:38.776232 3550 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:25:38.783085 3550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2c7f8df5f4d4f90b0f195b0e3c59d5f: Generated new TSK 0
I20260811 22:25:38.784020 3550 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.106.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab.
Entry for principal kudu/127.29.106.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab.
I20260811 22:25:38.831456 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.1:0
--local_ip_for_outbound_sockets=127.29.106.1
--webserver_interface=127.29.106.1
--webserver_port=0
--tserver_master_addrs=127.29.106.62:45339
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab
--principal=kudu/127.29.106.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:35393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:25:39.146204 3569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:39.146469 3569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:39.146602 3569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:39.153680 3569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:39.153796 3569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:39.153955 3569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.1
I20260811 22:25:39.163349 3569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:35393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab
--principal=kudu/127.29.106.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.106.1
--webserver_port=0
--tserver_master_addrs=127.29.106.62:45339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.1
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:25:39.164539 3569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:39.165808 3569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:39.178076 3580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:39.178104 3583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:39.178540 3590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:39.178871 3569 server_base.cc:1034] running on GCE node
I20260811 22:25:39.179451 3569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:39.180351 3569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:39.181555 3569 hybrid_clock.cc:648] HybridClock initialized: now 1786487139181508 us; error 43 us; skew 500 ppm
I20260811 22:25:39.181842 3569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:25:39 dist-test-slave-gjrz krb5kdc[3363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487139, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:39.184864 3569 init.cc:377] Logged in from keytab as kudu/127.29.106.1@KRBTEST.COM (short username kudu)
I20260811 22:25:39.186204 3569 webserver.cc:460] Webserver started at http://127.29.106.1:45577/ using document root <none> and password file <none>
I20260811 22:25:39.186827 3569 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:39.186936 3569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:39.187227 3569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:39.190011 3569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-0/data/instance:
uuid: "f3eff03e105f424c9757f4a4c23703c8"
format_stamp: "Formatted at 2026-08-11 22:25:39 on dist-test-slave-gjrz"
server_key: "1439b178424c507f4cf465b623cdd34b"
server_key_iv: "50bf96764795c2e832b05992896f7537"
server_key_version: "encryptionkey@0"
I20260811 22:25:39.190730 3569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance:
uuid: "f3eff03e105f424c9757f4a4c23703c8"
format_stamp: "Formatted at 2026-08-11 22:25:39 on dist-test-slave-gjrz"
server_key: "1439b178424c507f4cf465b623cdd34b"
server_key_iv: "50bf96764795c2e832b05992896f7537"
server_key_version: "encryptionkey@0"
I20260811 22:25:39.195547 3569 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260811 22:25:39.198760 3597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:39.200129 3569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:25:39.200260 3569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-0/wal
uuid: "f3eff03e105f424c9757f4a4c23703c8"
format_stamp: "Formatted at 2026-08-11 22:25:39 on dist-test-slave-gjrz"
server_key: "1439b178424c507f4cf465b623cdd34b"
server_key_iv: "50bf96764795c2e832b05992896f7537"
server_key_version: "encryptionkey@0"
I20260811 22:25:39.200416 3569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:39.226990 3569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:39.230863 3569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:39.231118 3569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:39.232131 3569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:25:39.233393 3569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:25:39.233464 3569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:39.233551 3569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:25:39.233593 3569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:39.262861 3569 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.1:41161
I20260811 22:25:39.262919 3830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.1:41161 every 8 connection(s)
I20260811 22:25:39.264063 3569 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
I20260811 22:25:39.269687 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 3569
I20260811 22:25:39.269876 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance
I20260811 22:25:39.270462 30120 external_mini_cluster.cc:1428] Setting key 3e139b5268667a5566de4f9c09e7f961
Aug 11 22:25:39 dist-test-slave-gjrz krb5kdc[3363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487139, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.1@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:25:39.283401 3839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:39.266348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:59601 (local address 127.29.106.62:45339)
0811 22:25:39.266782 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:39.266786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:39.267604 (+ 818us) server_negotiation.cc:407] Connection header received
0811 22:25:39.269371 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:39.269377 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:39.269466 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:39.269559 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:39.271045 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:39.271791 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:39.273411 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:39.274026 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:39.274142 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:39.277571 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:39.277577 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:39.277580 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:39.277617 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:39.280186 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:39.281007 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:39.281010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:39.281013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:39.281126 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:39.281578 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:39.281582 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:39.281584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:39.281862 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:39.282005 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:39.282884 (+ 879us) server_negotiation.cc:299] Negotiation successful
0811 22:25:39.283053 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":238,"threads_started":1}
I20260811 22:25:39.284590 3843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:39.266735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45339 (local address 127.29.106.1:59601)
0811 22:25:39.267457 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:39.267501 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:39.269111 (+ 1610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:39.269790 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:39.269798 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:39.270310 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:39.270870 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:39.270887 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:39.271944 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:39.271950 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:39.273260 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:39.273277 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:39.274252 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:39.274257 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:39.274451 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:39.275062 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:39.275083 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:39.277349 (+ 2266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:39.280319 (+ 2970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:39.280328 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:39.280343 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:39.280896 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:39.281218 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:39.281223 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:39.281227 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:39.281452 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:39.282020 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:39.282028 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:39.282467 (+ 439us) client_negotiation.cc:769] Sending connection context
0811 22:25:39.282768 (+ 301us) client_negotiation.cc:240] Negotiation successful
0811 22:25:39.283257 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":267,"threads_started":1}
I20260811 22:25:39.286147 3837 heartbeater.cc:347] Connected to a master server at 127.29.106.62:45339
I20260811 22:25:39.286445 3837 heartbeater.cc:464] Registering TS with master...
I20260811 22:25:39.287318 3837 heartbeater.cc:511] Master 127.29.106.62:45339 requested a full tablet report, sending...
I20260811 22:25:39.289698 3448 ts_manager.cc:194] Registered new tserver with Master: f3eff03e105f424c9757f4a4c23703c8 (127.29.106.1:41161)
I20260811 22:25:39.291729 3448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.1@KRBTEST.COM'} at 127.29.106.1:59601
WARNING: no policy specified for kudu/127.29.106.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab.
Entry for principal kudu/127.29.106.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab.
I20260811 22:25:39.317772 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.2:0
--local_ip_for_outbound_sockets=127.29.106.2
--webserver_interface=127.29.106.2
--webserver_port=0
--tserver_master_addrs=127.29.106.62:45339
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab
--principal=kudu/127.29.106.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:35393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:25:39.627851 3848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:39.628104 3848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:39.628229 3848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:39.635221 3848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:39.635339 3848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:39.635459 3848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.2
I20260811 22:25:39.644567 3848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:35393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab
--principal=kudu/127.29.106.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.106.2
--webserver_port=0
--tserver_master_addrs=127.29.106.62:45339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.2
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:25:39.645727 3848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:39.647001 3848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:39.658943 3858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:39.659034 3859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:39.659073 3861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:39.660151 3848 server_base.cc:1034] running on GCE node
I20260811 22:25:39.660692 3848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:39.661518 3848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:39.662724 3848 hybrid_clock.cc:648] HybridClock initialized: now 1786487139662670 us; error 46 us; skew 500 ppm
I20260811 22:25:39.663002 3848 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:25:39 dist-test-slave-gjrz krb5kdc[3363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487139, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:39.666029 3848 init.cc:377] Logged in from keytab as kudu/127.29.106.2@KRBTEST.COM (short username kudu)
I20260811 22:25:39.667351 3848 webserver.cc:460] Webserver started at http://127.29.106.2:45945/ using document root <none> and password file <none>
I20260811 22:25:39.667943 3848 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:39.668067 3848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:39.668341 3848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:39.671093 3848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-1/data/instance:
uuid: "c7401c557e7c49d78927104ea22b6ab0"
format_stamp: "Formatted at 2026-08-11 22:25:39 on dist-test-slave-gjrz"
server_key: "6bf19accf0ac2d072623afa9cf187930"
server_key_iv: "12fda03a782b280bdbbbd52e8c00c1b3"
server_key_version: "encryptionkey@0"
I20260811 22:25:39.671855 3848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance:
uuid: "c7401c557e7c49d78927104ea22b6ab0"
format_stamp: "Formatted at 2026-08-11 22:25:39 on dist-test-slave-gjrz"
server_key: "6bf19accf0ac2d072623afa9cf187930"
server_key_iv: "12fda03a782b280bdbbbd52e8c00c1b3"
server_key_version: "encryptionkey@0"
I20260811 22:25:39.676630 3848 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260811 22:25:39.679919 3881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:39.681169 3848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 22:25:39.681316 3848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-1/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-1/wal
uuid: "c7401c557e7c49d78927104ea22b6ab0"
format_stamp: "Formatted at 2026-08-11 22:25:39 on dist-test-slave-gjrz"
server_key: "6bf19accf0ac2d072623afa9cf187930"
server_key_iv: "12fda03a782b280bdbbbd52e8c00c1b3"
server_key_version: "encryptionkey@0"
I20260811 22:25:39.681494 3848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:39.696197 3848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:39.701120 3848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:39.701407 3848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:39.702507 3848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:25:39.704128 3848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:25:39.704223 3848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:39.704344 3848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:25:39.704406 3848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:39.734625 3848 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.2:45347
I20260811 22:25:39.734686 4091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.2:45347 every 8 connection(s)
I20260811 22:25:39.735823 3848 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
I20260811 22:25:39.743826 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 3848
I20260811 22:25:39.743966 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance
I20260811 22:25:39.744380 30120 external_mini_cluster.cc:1428] Setting key 41dbb0e6da86072d0c098583e532531a
Aug 11 22:25:39 dist-test-slave-gjrz krb5kdc[3363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487139, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.2@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:25:39.753996 3839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:39.738070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:43879 (local address 127.29.106.62:45339)
0811 22:25:39.738275 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:39.738282 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:39.739354 (+ 1072us) server_negotiation.cc:407] Connection header received
0811 22:25:39.741043 (+ 1689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:39.741047 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:39.741113 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:39.741196 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:39.742599 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:39.743337 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:39.744902 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:39.745511 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:39.745633 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:39.748821 (+ 3188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:39.748827 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:39.748830 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:39.748860 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:39.751208 (+ 2348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:39.751878 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:39.751882 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:39.751885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:39.752006 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:39.752395 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:39.752407 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:39.752409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:39.752669 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:39.752802 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:39.753449 (+ 647us) server_negotiation.cc:299] Negotiation successful
0811 22:25:39.753687 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260811 22:25:39.754815 4099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:39.738504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45339 (local address 127.29.106.2:43879)
0811 22:25:39.739191 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:39.739226 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:39.740804 (+ 1578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:39.741384 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:39.741394 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:39.741957 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:39.742470 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:39.742486 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:39.743464 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:39.743468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:39.744763 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:39.744775 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:39.745719 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:39.745727 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:39.745948 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:39.746571 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:39.746592 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:39.748627 (+ 2035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:39.751355 (+ 2728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:39.751363 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:39.751378 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:39.751766 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:39.752107 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:39.752112 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:39.752116 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:39.752289 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:39.752789 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:39.752797 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:39.753091 (+ 294us) client_negotiation.cc:769] Sending connection context
0811 22:25:39.753317 (+ 226us) client_negotiation.cc:240] Negotiation successful
0811 22:25:39.753779 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":256,"threads_started":1}
I20260811 22:25:39.756117 4092 heartbeater.cc:347] Connected to a master server at 127.29.106.62:45339
I20260811 22:25:39.756424 4092 heartbeater.cc:464] Registering TS with master...
I20260811 22:25:39.757293 4092 heartbeater.cc:511] Master 127.29.106.62:45339 requested a full tablet report, sending...
I20260811 22:25:39.758891 3448 ts_manager.cc:194] Registered new tserver with Master: c7401c557e7c49d78927104ea22b6ab0 (127.29.106.2:45347)
I20260811 22:25:39.760006 3448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.2@KRBTEST.COM'} at 127.29.106.2:43879
WARNING: no policy specified for kudu/127.29.106.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab.
Entry for principal kudu/127.29.106.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab.
I20260811 22:25:39.789816 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.3:0
--local_ip_for_outbound_sockets=127.29.106.3
--webserver_interface=127.29.106.3
--webserver_port=0
--tserver_master_addrs=127.29.106.62:45339
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab
--principal=kudu/127.29.106.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:35393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:25:40.099084 4105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:40.099370 4105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:40.099481 4105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:40.106526 4105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:40.106617 4105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:40.106732 4105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.3
I20260811 22:25:40.115821 4105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:35393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab
--principal=kudu/127.29.106.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.106.3
--webserver_port=0
--tserver_master_addrs=127.29.106.62:45339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.3
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:25:40.116885 4105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:40.118086 4105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:40.130367 4117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:40.130367 4115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:40.130403 4114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:40.131374 4105 server_base.cc:1034] running on GCE node
I20260811 22:25:40.131966 4105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:40.132884 4105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:40.134085 4105 hybrid_clock.cc:648] HybridClock initialized: now 1786487140134034 us; error 38 us; skew 500 ppm
I20260811 22:25:40.134398 4105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:25:40 dist-test-slave-gjrz krb5kdc[3363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487140, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:40.137461 4105 init.cc:377] Logged in from keytab as kudu/127.29.106.3@KRBTEST.COM (short username kudu)
I20260811 22:25:40.138825 4105 webserver.cc:460] Webserver started at http://127.29.106.3:46135/ using document root <none> and password file <none>
I20260811 22:25:40.139556 4105 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:40.139674 4105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:40.139954 4105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:40.142803 4105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-2/data/instance:
uuid: "c6f050c8638043d69a2fc1d7a5da43bc"
format_stamp: "Formatted at 2026-08-11 22:25:40 on dist-test-slave-gjrz"
server_key: "c7391a9f6d782d6a7da137db2c5d8310"
server_key_iv: "6d942e3b0ef917549d40e4f3df68eccc"
server_key_version: "encryptionkey@0"
I20260811 22:25:40.143620 4105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance:
uuid: "c6f050c8638043d69a2fc1d7a5da43bc"
format_stamp: "Formatted at 2026-08-11 22:25:40 on dist-test-slave-gjrz"
server_key: "c7391a9f6d782d6a7da137db2c5d8310"
server_key_iv: "6d942e3b0ef917549d40e4f3df68eccc"
server_key_version: "encryptionkey@0"
I20260811 22:25:40.148469 4105 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260811 22:25:40.151749 4128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:40.152976 4105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260811 22:25:40.153142 4105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-2/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-2/wal
uuid: "c6f050c8638043d69a2fc1d7a5da43bc"
format_stamp: "Formatted at 2026-08-11 22:25:40 on dist-test-slave-gjrz"
server_key: "c7391a9f6d782d6a7da137db2c5d8310"
server_key_iv: "6d942e3b0ef917549d40e4f3df68eccc"
server_key_version: "encryptionkey@0"
I20260811 22:25:40.153322 4105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:40.172518 4105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:40.176364 4105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:40.176628 4105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:40.177577 4105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:25:40.178848 4105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:25:40.178928 4105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:40.179056 4105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:25:40.179109 4105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:40.208771 4105 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.3:43745
I20260811 22:25:40.208832 4344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.3:43745 every 8 connection(s)
I20260811 22:25:40.209966 4105 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
I20260811 22:25:40.215690 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 4105
I20260811 22:25:40.215871 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance
I20260811 22:25:40.216326 30120 external_mini_cluster.cc:1428] Setting key ed1330b547520740578b1df10677a93a
Aug 11 22:25:40 dist-test-slave-gjrz krb5kdc[3363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487140, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.3@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:25:40.228355 3839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:40.212437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:42637 (local address 127.29.106.62:45339)
0811 22:25:40.212601 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:40.212606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:40.213724 (+ 1118us) server_negotiation.cc:407] Connection header received
0811 22:25:40.215525 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:40.215530 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:40.215627 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:40.215742 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:40.217301 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:40.218204 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:40.219839 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:40.220245 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:40.220337 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:40.223440 (+ 3103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:40.223445 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:40.223449 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:40.223482 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:40.225964 (+ 2482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:40.226572 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:40.226575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:40.226577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:40.226699 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:40.227021 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:40.227024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:40.227026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:40.227313 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:40.227421 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:40.227956 (+ 535us) server_negotiation.cc:299] Negotiation successful
0811 22:25:40.228119 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260811 22:25:40.229203 4350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:40.212877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45339 (local address 127.29.106.3:42637)
0811 22:25:40.213573 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:40.213615 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:40.215229 (+ 1614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:40.215956 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:40.215968 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:40.216517 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:40.217122 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:40.217147 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:40.218350 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:40.218354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:40.219716 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:40.219730 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:40.220419 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:40.220424 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:40.220614 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:40.221136 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:40.221161 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:40.223231 (+ 2070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:40.226073 (+ 2842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:40.226080 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:40.226091 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:40.226472 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:40.226775 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:40.226778 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:40.226780 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:40.226943 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:40.227397 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:40.227402 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:40.227675 (+ 273us) client_negotiation.cc:769] Sending connection context
0811 22:25:40.227880 (+ 205us) client_negotiation.cc:240] Negotiation successful
0811 22:25:40.228243 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":269,"threads_started":1}
I20260811 22:25:40.230525 4345 heartbeater.cc:347] Connected to a master server at 127.29.106.62:45339
I20260811 22:25:40.230852 4345 heartbeater.cc:464] Registering TS with master...
I20260811 22:25:40.231670 4345 heartbeater.cc:511] Master 127.29.106.62:45339 requested a full tablet report, sending...
I20260811 22:25:40.232914 3447 ts_manager.cc:194] Registered new tserver with Master: c6f050c8638043d69a2fc1d7a5da43bc (127.29.106.3:43745)
I20260811 22:25:40.233701 3447 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.3@KRBTEST.COM'} at 127.29.106.3:42637
I20260811 22:25:40.243355 30120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 11 22:25:40 dist-test-slave-gjrz krb5kdc[3363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487140, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 11 22:25:40 dist-test-slave-gjrz krb5kdc[3363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487140, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:25:40.294600 3837 heartbeater.cc:503] Master 127.29.106.62:45339 was elected leader, sending a full tablet report...
I20260811 22:25:40.294827 3839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:40.281401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56532 (local address 127.29.106.62:45339)
0811 22:25:40.281614 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:40.281621 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:40.281981 (+ 360us) server_negotiation.cc:407] Connection header received
0811 22:25:40.282171 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:40.282175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:40.282241 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:40.282325 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:40.283678 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:40.284453 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:40.285795 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:40.286204 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:40.286311 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:40.289864 (+ 3553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:40.289869 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:40.289872 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:40.289905 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:40.292312 (+ 2407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:40.292902 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:40.292906 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:40.292909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:40.293045 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:40.293402 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:40.293407 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:40.293410 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:40.293728 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:40.293967 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:40.294259 (+ 292us) server_negotiation.cc:299] Negotiation successful
0811 22:25:40.294503 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260811 22:25:40.295462 3447 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:56532
I20260811 22:25:40.297045 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 3569
Aug 11 22:25:40 dist-test-slave-gjrz krb5kdc[3363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487140, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.29.106.1@KRBTEST.COM
I20260811 22:25:40.312146 4364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:40.298134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60996 (local address 127.29.106.1:41161)
0811 22:25:40.298727 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:40.298734 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:40.298758 (+ 24us) server_negotiation.cc:407] Connection header received
0811 22:25:40.298807 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:40.298813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:40.299012 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:40.299130 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:40.300271 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:40.301323 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:40.302500 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:40.303061 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:40.303162 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:40.307018 (+ 3856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:40.307032 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:40.307048 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:40.307084 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:40.309626 (+ 2542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:40.310215 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:40.310222 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:40.310224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:40.310349 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:40.310689 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:40.310692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:40.310694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:40.311015 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:40.311330 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:40.311603 (+ 273us) server_negotiation.cc:299] Negotiation successful
0811 22:25:40.311843 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":306,"threads_started":1}
W20260811 22:25:40.313330 3818 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:60996
W20260811 22:25:40.314364 30120 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 3569: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 22:25:40.314563 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 3569
I20260811 22:25:40.340456 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 3848
Aug 11 22:25:40 dist-test-slave-gjrz krb5kdc[3363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487140, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.29.106.2@KRBTEST.COM
I20260811 22:25:40.355165 4365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:40.341257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41140 (local address 127.29.106.2:45347)
0811 22:25:40.341791 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:40.341797 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:40.341821 (+ 24us) server_negotiation.cc:407] Connection header received
0811 22:25:40.341910 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:40.341916 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:40.342132 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:40.342279 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:40.343522 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:40.344537 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:40.345844 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:40.346396 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:40.346510 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:40.350400 (+ 3890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:40.350411 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:40.350425 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:40.350457 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:40.352991 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:40.353534 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:40.353541 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:40.353543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:40.353668 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:40.353915 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:40.353918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:40.353920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:40.354224 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:40.354459 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:40.354687 (+ 228us) server_negotiation.cc:299] Negotiation successful
0811 22:25:40.354879 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":240,"threads_started":1}
W20260811 22:25:40.356395 4081 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41140
W20260811 22:25:40.357445 30120 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 3848: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 22:25:40.357578 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 3848
I20260811 22:25:40.381943 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 4105
Aug 11 22:25:40 dist-test-slave-gjrz krb5kdc[3363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487140, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.29.106.3@KRBTEST.COM
I20260811 22:25:40.397161 4366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:40.382862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53438 (local address 127.29.106.3:43745)
0811 22:25:40.383542 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:40.383561 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:40.383590 (+ 29us) server_negotiation.cc:407] Connection header received
0811 22:25:40.383652 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:40.383658 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:40.383858 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:40.383984 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:40.385186 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:40.386203 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:40.387370 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:40.387920 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:40.388016 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:40.392035 (+ 4019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:40.392047 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:40.392061 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:40.392097 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:40.394491 (+ 2394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:40.395110 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:40.395117 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:40.395120 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:40.395244 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:40.395619 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:40.395624 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:40.395628 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:40.395961 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:40.396251 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:40.396569 (+ 318us) server_negotiation.cc:299] Negotiation successful
0811 22:25:40.396832 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":370,"threads_started":1}
W20260811 22:25:40.398355 4339 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53438
W20260811 22:25:40.399398 30120 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 4105: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 22:25:40.399569 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 4105
I20260811 22:25:40.420100 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 3383
I20260811 22:25:40.421139 3508 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:25:40.522805 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 3383
2026-08-11T22:25:40Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3678 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:25:40 dist-test-slave-gjrz krb5kdc[4372](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:25:40 dist-test-slave-gjrz krb5kdc[4372](info): set up 2 sockets
Aug 11 22:25:40 dist-test-slave-gjrz krb5kdc[4372](info): commencing operation
krb5kdc: starting...
W20260811 22:25:41.782519 30120 mini_kdc.cc:121] Time spent starting KDC: real 1.209s user 0.001s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Aug 11 22:25:41 dist-test-slave-gjrz krb5kdc[4372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:25:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:25:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.106.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
I20260811 22:25:41.972239 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:38813
--webserver_interface=127.29.106.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:37219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:38813
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:25:42.303536 4395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:42.303848 4395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:42.304046 4395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:42.311694 4395 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:25:42.311790 4395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:42.311843 4395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:42.311913 4395 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:25:42.311957 4395 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:25:42.322117 4395 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:37219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:38813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:38813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.106.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:25:42.323480 4395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:42.324813 4395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:42.336503 4400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:42.336589 4414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:42.336583 4408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:42.337536 4395 server_base.cc:1034] running on GCE node
I20260811 22:25:42.338277 4395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:42.339701 4395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:42.340914 4395 hybrid_clock.cc:648] HybridClock initialized: now 1786487142340859 us; error 37 us; skew 500 ppm
I20260811 22:25:42.341241 4395 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:25:42 dist-test-slave-gjrz krb5kdc[4372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487142, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:42.344308 4395 init.cc:377] Logged in from keytab as kudu/127.29.106.62@KRBTEST.COM (short username kudu)
I20260811 22:25:42.345662 4395 webserver.cc:460] Webserver started at http://127.29.106.62:46611/ using document root <none> and password file <none>
I20260811 22:25:42.346310 4395 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:42.346408 4395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:42.346647 4395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:42.349736 4395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "fe495d19ec5246fd8a59ccfbe8522147"
format_stamp: "Formatted at 2026-08-11 22:25:42 on dist-test-slave-gjrz"
server_key: "2e4cd00e0cbd29e951d19438741152f8"
server_key_iv: "7c62ba814aebcd49e9be3d7c83c1a2cc"
server_key_version: "encryptionkey@0"
I20260811 22:25:42.350492 4395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "fe495d19ec5246fd8a59ccfbe8522147"
format_stamp: "Formatted at 2026-08-11 22:25:42 on dist-test-slave-gjrz"
server_key: "2e4cd00e0cbd29e951d19438741152f8"
server_key_iv: "7c62ba814aebcd49e9be3d7c83c1a2cc"
server_key_version: "encryptionkey@0"
I20260811 22:25:42.355453 4395 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260811 22:25:42.358742 4428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:42.360000 4395 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:25:42.360174 4395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "fe495d19ec5246fd8a59ccfbe8522147"
format_stamp: "Formatted at 2026-08-11 22:25:42 on dist-test-slave-gjrz"
server_key: "2e4cd00e0cbd29e951d19438741152f8"
server_key_iv: "7c62ba814aebcd49e9be3d7c83c1a2cc"
server_key_version: "encryptionkey@0"
I20260811 22:25:42.360342 4395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:42.376698 4395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:42.380347 4395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:42.380594 4395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:42.398144 4395 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.62:38813
I20260811 22:25:42.398169 4516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.62:38813 every 8 connection(s)
I20260811 22:25:42.399394 4395 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:25:42.399845 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 4395
I20260811 22:25:42.400102 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:25:42.400846 30120 external_mini_cluster.cc:1428] Setting key 0466fa24269703c37bfbbe125e3b78d2
I20260811 22:25:42.403541 4518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 11 22:25:42 dist-test-slave-gjrz krb5kdc[4372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:25:42.416527 4518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe495d19ec5246fd8a59ccfbe8522147: Bootstrap starting.
I20260811 22:25:42.420521 4518 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe495d19ec5246fd8a59ccfbe8522147: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:42.421694 4518 log.cc:824] T 00000000000000000000000000000000 P fe495d19ec5246fd8a59ccfbe8522147: Log is configured to *not* fsync() on all Append() calls
I20260811 22:25:42.429004 4521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:42.402644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57552 (local address 127.29.106.62:38813)
0811 22:25:42.403399 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:42.403410 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:42.403451 (+ 41us) server_negotiation.cc:407] Connection header received
0811 22:25:42.405089 (+ 1638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:42.405114 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:42.405451 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:42.405755 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:42.407508 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:42.408852 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:42.410664 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:42.411689 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:42.411854 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:42.416523 (+ 4669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:42.416534 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:42.416561 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:42.416612 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:42.420330 (+ 3718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:42.420976 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:42.420984 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:42.420991 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:42.421196 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:42.421524 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:42.421528 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:42.421531 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:42.422236 (+ 705us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:42.422611 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:42.425691 (+ 3080us) server_negotiation.cc:299] Negotiation successful
0811 22:25:42.426417 (+ 726us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":307,"threads_started":1}
I20260811 22:25:42.429788 4518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe495d19ec5246fd8a59ccfbe8522147: No bootstrap required, opened a new log
I20260811 22:25:42.441689 4518 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe495d19ec5246fd8a59ccfbe8522147 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe495d19ec5246fd8a59ccfbe8522147" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 38813 } }
I20260811 22:25:42.442107 4518 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe495d19ec5246fd8a59ccfbe8522147 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:25:42.442238 4518 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe495d19ec5246fd8a59ccfbe8522147 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe495d19ec5246fd8a59ccfbe8522147, State: Initialized, Role: FOLLOWER
I20260811 22:25:42.442852 4518 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe495d19ec5246fd8a59ccfbe8522147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe495d19ec5246fd8a59ccfbe8522147" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 38813 } }
I20260811 22:25:42.443228 4518 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe495d19ec5246fd8a59ccfbe8522147 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:25:42.443460 4518 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe495d19ec5246fd8a59ccfbe8522147 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:25:42.443672 4518 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe495d19ec5246fd8a59ccfbe8522147 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:42.446563 4518 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe495d19ec5246fd8a59ccfbe8522147 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe495d19ec5246fd8a59ccfbe8522147" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 38813 } }
I20260811 22:25:42.447089 4518 leader_election.cc:302] T 00000000000000000000000000000000 P fe495d19ec5246fd8a59ccfbe8522147 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe495d19ec5246fd8a59ccfbe8522147; no voters:
I20260811 22:25:42.447692 4518 leader_election.cc:288] T 00000000000000000000000000000000 P fe495d19ec5246fd8a59ccfbe8522147 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:25:42.447810 4528 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe495d19ec5246fd8a59ccfbe8522147 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:25:42.448556 4528 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe495d19ec5246fd8a59ccfbe8522147 [term 1 LEADER]: Becoming Leader. State: Replica: fe495d19ec5246fd8a59ccfbe8522147, State: Running, Role: LEADER
I20260811 22:25:42.449066 4528 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe495d19ec5246fd8a59ccfbe8522147 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe495d19ec5246fd8a59ccfbe8522147" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 38813 } }
I20260811 22:25:42.449792 4518 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe495d19ec5246fd8a59ccfbe8522147 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:25:42.451733 4531 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe495d19ec5246fd8a59ccfbe8522147 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe495d19ec5246fd8a59ccfbe8522147" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe495d19ec5246fd8a59ccfbe8522147" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 38813 } } }
I20260811 22:25:42.452209 4531 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe495d19ec5246fd8a59ccfbe8522147 [sys.catalog]: This master's current role is: LEADER
I20260811 22:25:42.452023 4532 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe495d19ec5246fd8a59ccfbe8522147 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe495d19ec5246fd8a59ccfbe8522147. Latest consensus state: current_term: 1 leader_uuid: "fe495d19ec5246fd8a59ccfbe8522147" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe495d19ec5246fd8a59ccfbe8522147" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 38813 } } }
I20260811 22:25:42.452330 4532 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe495d19ec5246fd8a59ccfbe8522147 [sys.catalog]: This master's current role is: LEADER
I20260811 22:25:42.454450 4541 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:25:42.458949 4541 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:25:42.467017 4541 catalog_manager.cc:1347] Generated new cluster ID: 710a053ee08f4e9ea9d9ee3b925f2b90
I20260811 22:25:42.467126 4541 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:25:42.479122 4541 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:25:42.480391 4541 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:25:42.489351 4541 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe495d19ec5246fd8a59ccfbe8522147: Generated new TSK 0
I20260811 22:25:42.490060 4541 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.106.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab.
Entry for principal kudu/127.29.106.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab.
W20260811 22:25:42.685496 30120 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.29.106.1: real 0.188s user 0.000s sys 0.052s
I20260811 22:25:42.686462 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.1:0
--local_ip_for_outbound_sockets=127.29.106.1
--webserver_interface=127.29.106.1
--webserver_port=0
--tserver_master_addrs=127.29.106.62:38813
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab
--principal=kudu/127.29.106.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:37219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:25:43.047648 4566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:43.047953 4566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:43.048062 4566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:43.056064 4566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:43.056169 4566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:43.056294 4566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.1
I20260811 22:25:43.066287 4566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:37219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab
--principal=kudu/127.29.106.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.106.1
--webserver_port=0
--tserver_master_addrs=127.29.106.62:38813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.1
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:25:43.067497 4566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:43.068718 4566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:43.081188 4582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:43.081164 4580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:43.081502 4574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:43.081974 4566 server_base.cc:1034] running on GCE node
I20260811 22:25:43.082499 4566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:43.083402 4566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:43.084611 4566 hybrid_clock.cc:648] HybridClock initialized: now 1786487143084555 us; error 46 us; skew 500 ppm
I20260811 22:25:43.084888 4566 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:25:43 dist-test-slave-gjrz krb5kdc[4372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487143, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:43.088038 4566 init.cc:377] Logged in from keytab as kudu/127.29.106.1@KRBTEST.COM (short username kudu)
I20260811 22:25:43.089310 4566 webserver.cc:460] Webserver started at http://127.29.106.1:32927/ using document root <none> and password file <none>
I20260811 22:25:43.089892 4566 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:43.090000 4566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:43.090242 4566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:43.093083 4566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/ts-0/data/instance:
uuid: "6d9eea98628f49718ae4388fa84e5151"
format_stamp: "Formatted at 2026-08-11 22:25:43 on dist-test-slave-gjrz"
server_key: "b849c9cb5332bd8de26bc939adde2f8a"
server_key_iv: "ebd97a2d405290518690de213f126f2b"
server_key_version: "encryptionkey@0"
I20260811 22:25:43.093806 4566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance:
uuid: "6d9eea98628f49718ae4388fa84e5151"
format_stamp: "Formatted at 2026-08-11 22:25:43 on dist-test-slave-gjrz"
server_key: "b849c9cb5332bd8de26bc939adde2f8a"
server_key_iv: "ebd97a2d405290518690de213f126f2b"
server_key_version: "encryptionkey@0"
I20260811 22:25:43.098630 4566 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260811 22:25:43.102005 4596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:43.103354 4566 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:25:43.103503 4566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/ts-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/ts-0/wal
uuid: "6d9eea98628f49718ae4388fa84e5151"
format_stamp: "Formatted at 2026-08-11 22:25:43 on dist-test-slave-gjrz"
server_key: "b849c9cb5332bd8de26bc939adde2f8a"
server_key_iv: "ebd97a2d405290518690de213f126f2b"
server_key_version: "encryptionkey@0"
I20260811 22:25:43.103685 4566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:43.115183 4566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:43.118963 4566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:43.119220 4566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:43.120182 4566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:25:43.121444 4566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:25:43.121518 4566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:43.121640 4566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:25:43.121695 4566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:43.152796 4566 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.1:36455
I20260811 22:25:43.152868 4820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.1:36455 every 8 connection(s)
I20260811 22:25:43.154232 4566 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
I20260811 22:25:43.157967 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 4566
I20260811 22:25:43.158196 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance
I20260811 22:25:43.159065 30120 external_mini_cluster.cc:1428] Setting key 9263e3e1791897a7c841e31387f405a0
Aug 11 22:25:43 dist-test-slave-gjrz krb5kdc[4372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487143, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.1@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:25:43.175526 4824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:43.156818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:59473 (local address 127.29.106.62:38813)
0811 22:25:43.157292 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:43.157296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:43.158303 (+ 1007us) server_negotiation.cc:407] Connection header received
0811 22:25:43.160657 (+ 2354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:43.160662 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:43.160729 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:43.160823 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:43.162766 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:43.163633 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:43.165781 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:43.166186 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:43.166281 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:43.170025 (+ 3744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:43.170030 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:43.170033 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:43.170065 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:43.172539 (+ 2474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:43.173222 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:43.173226 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:43.173228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:43.173348 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:43.173761 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:43.173766 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:43.173769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:43.174055 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:43.174187 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:43.174868 (+ 681us) server_negotiation.cc:299] Negotiation successful
0811 22:25:43.175130 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":250,"threads_started":1}
I20260811 22:25:43.176605 4830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:43.157260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:38813 (local address 127.29.106.1:59473)
0811 22:25:43.158104 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:43.158156 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:43.160328 (+ 2172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:43.161072 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:43.161082 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:43.161775 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:43.162583 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:43.162609 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:43.163792 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:43.163798 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:43.165646 (+ 1848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:43.165666 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:43.166422 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:43.166429 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:43.166700 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:43.167390 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:43.167416 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:43.169811 (+ 2395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:43.172672 (+ 2861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:43.172682 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:43.172699 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:43.173107 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:43.173459 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:43.173464 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:43.173468 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:43.173652 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:43.174178 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:43.174187 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:43.174508 (+ 321us) client_negotiation.cc:769] Sending connection context
0811 22:25:43.174753 (+ 245us) client_negotiation.cc:240] Negotiation successful
0811 22:25:43.175256 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":300,"threads_started":1}
I20260811 22:25:43.178319 4821 heartbeater.cc:347] Connected to a master server at 127.29.106.62:38813
I20260811 22:25:43.178618 4821 heartbeater.cc:464] Registering TS with master...
I20260811 22:25:43.179358 4821 heartbeater.cc:511] Master 127.29.106.62:38813 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.29.106.2@KRBTEST.COM; defaulting to no policy
I20260811 22:25:43.181370 4454 ts_manager.cc:194] Registered new tserver with Master: 6d9eea98628f49718ae4388fa84e5151 (127.29.106.1:36455)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.2@KRBTEST.COM" created.
I20260811 22:25:43.182921 4454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.1@KRBTEST.COM'} at 127.29.106.1:59473
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab.
Entry for principal kudu/127.29.106.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab.
I20260811 22:25:43.206368 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.2:0
--local_ip_for_outbound_sockets=127.29.106.2
--webserver_interface=127.29.106.2
--webserver_port=0
--tserver_master_addrs=127.29.106.62:38813
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab
--principal=kudu/127.29.106.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:37219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:25:43.530648 4837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:43.530925 4837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:43.531055 4837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:43.538198 4837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:43.538296 4837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:43.538448 4837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.2
I20260811 22:25:43.547859 4837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:37219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab
--principal=kudu/127.29.106.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.106.2
--webserver_port=0
--tserver_master_addrs=127.29.106.62:38813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.2
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:25:43.549005 4837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:43.550246 4837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:43.562649 4847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:43.562865 4844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:43.562690 4854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:43.563575 4837 server_base.cc:1034] running on GCE node
I20260811 22:25:43.564129 4837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:43.564951 4837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:43.566143 4837 hybrid_clock.cc:648] HybridClock initialized: now 1786487143566103 us; error 32 us; skew 500 ppm
I20260811 22:25:43.566425 4837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:25:43 dist-test-slave-gjrz krb5kdc[4372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487143, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:43.569315 4837 init.cc:377] Logged in from keytab as kudu/127.29.106.2@KRBTEST.COM (short username kudu)
I20260811 22:25:43.570667 4837 webserver.cc:460] Webserver started at http://127.29.106.2:33243/ using document root <none> and password file <none>
I20260811 22:25:43.571300 4837 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:43.571413 4837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:43.571676 4837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:43.574479 4837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/ts-1/data/instance:
uuid: "58b97bbd5f2f429391cc72c97c5a8c97"
format_stamp: "Formatted at 2026-08-11 22:25:43 on dist-test-slave-gjrz"
server_key: "3c3b648c67909bfb78c0728a89178e94"
server_key_iv: "a8cdbb5f6d91549e6bbcb321247ca3d0"
server_key_version: "encryptionkey@0"
I20260811 22:25:43.575192 4837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance:
uuid: "58b97bbd5f2f429391cc72c97c5a8c97"
format_stamp: "Formatted at 2026-08-11 22:25:43 on dist-test-slave-gjrz"
server_key: "3c3b648c67909bfb78c0728a89178e94"
server_key_iv: "a8cdbb5f6d91549e6bbcb321247ca3d0"
server_key_version: "encryptionkey@0"
I20260811 22:25:43.580067 4837 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260811 22:25:43.583209 4870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:43.584417 4837 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260811 22:25:43.584546 4837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/ts-1/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/ts-1/wal
uuid: "58b97bbd5f2f429391cc72c97c5a8c97"
format_stamp: "Formatted at 2026-08-11 22:25:43 on dist-test-slave-gjrz"
server_key: "3c3b648c67909bfb78c0728a89178e94"
server_key_iv: "a8cdbb5f6d91549e6bbcb321247ca3d0"
server_key_version: "encryptionkey@0"
I20260811 22:25:43.584715 4837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:43.609695 4837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:43.613245 4837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:43.613488 4837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:43.614383 4837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:25:43.615649 4837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:25:43.615720 4837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:43.615831 4837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:25:43.615882 4837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:43.645457 4837 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.2:33107
I20260811 22:25:43.645505 5074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.2:33107 every 8 connection(s)
I20260811 22:25:43.646606 4837 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
I20260811 22:25:43.654477 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 4837
I20260811 22:25:43.654647 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance
I20260811 22:25:43.655131 30120 external_mini_cluster.cc:1428] Setting key 16114ea64dbab1d152ea58a0a33da4be
Aug 11 22:25:43 dist-test-slave-gjrz krb5kdc[4372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487143, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.2@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:25:43.673357 4824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:43.648917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:35709 (local address 127.29.106.62:38813)
0811 22:25:43.649076 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:43.649080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:43.650249 (+ 1169us) server_negotiation.cc:407] Connection header received
0811 22:25:43.652104 (+ 1855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:43.652109 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:43.652184 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:43.652274 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:43.653739 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:43.654699 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:43.656424 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:43.656973 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:43.657093 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:43.667725 (+ 10632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:43.667730 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:43.667734 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:43.667766 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:43.670213 (+ 2447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:43.671067 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:43.671071 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:43.671073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:43.671195 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:43.671673 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:43.671678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:43.671680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:43.671968 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:43.672083 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:43.672909 (+ 826us) server_negotiation.cc:299] Negotiation successful
0811 22:25:43.673084 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260811 22:25:43.674793 5079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:43.649342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:38813 (local address 127.29.106.2:35709)
0811 22:25:43.650078 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:43.650125 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:43.651824 (+ 1699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:43.652456 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:43.652463 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:43.652969 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:43.653566 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:43.653585 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:43.654848 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:43.654853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:43.656281 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:43.656294 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:43.657189 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:43.657195 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:43.657380 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:43.664834 (+ 7454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:43.664864 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:43.667513 (+ 2649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:43.670349 (+ 2836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:43.670359 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:43.670375 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:43.670965 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:43.671310 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:43.671315 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:43.671319 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:43.671558 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:43.672100 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:43.672109 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:43.672515 (+ 406us) client_negotiation.cc:769] Sending connection context
0811 22:25:43.672833 (+ 318us) client_negotiation.cc:240] Negotiation successful
0811 22:25:43.673359 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":267,"threads_started":1}
WARNING: no policy specified for kudu/127.29.106.3@KRBTEST.COM; defaulting to no policy
I20260811 22:25:43.676251 5077 heartbeater.cc:347] Connected to a master server at 127.29.106.62:38813
I20260811 22:25:43.676589 5077 heartbeater.cc:464] Registering TS with master...
I20260811 22:25:43.677363 5077 heartbeater.cc:511] Master 127.29.106.62:38813 requested a full tablet report, sending...
I20260811 22:25:43.678828 4454 ts_manager.cc:194] Registered new tserver with Master: 58b97bbd5f2f429391cc72c97c5a8c97 (127.29.106.2:33107)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.3@KRBTEST.COM" created.
I20260811 22:25:43.679914 4454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.2@KRBTEST.COM'} at 127.29.106.2:35709
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab.
Entry for principal kudu/127.29.106.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab.
I20260811 22:25:43.701959 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.3:0
--local_ip_for_outbound_sockets=127.29.106.3
--webserver_interface=127.29.106.3
--webserver_port=0
--tserver_master_addrs=127.29.106.62:38813
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab
--principal=kudu/127.29.106.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:37219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:25:44.013271 5088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:44.013541 5088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:44.013666 5088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:44.020648 5088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:44.020757 5088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:44.020916 5088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.3
I20260811 22:25:44.030059 5088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:37219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab
--principal=kudu/127.29.106.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.106.3
--webserver_port=0
--tserver_master_addrs=127.29.106.62:38813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.3
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:25:44.031173 5088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:44.032405 5088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:44.045246 5100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:44.045214 5106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:44.045174 5103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:44.045827 5088 server_base.cc:1034] running on GCE node
I20260811 22:25:44.046401 5088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:44.047312 5088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:44.048537 5088 hybrid_clock.cc:648] HybridClock initialized: now 1786487144048477 us; error 44 us; skew 500 ppm
I20260811 22:25:44.048908 5088 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:25:44 dist-test-slave-gjrz krb5kdc[4372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487144, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:44.052076 5088 init.cc:377] Logged in from keytab as kudu/127.29.106.3@KRBTEST.COM (short username kudu)
I20260811 22:25:44.053519 5088 webserver.cc:460] Webserver started at http://127.29.106.3:46413/ using document root <none> and password file <none>
I20260811 22:25:44.054167 5088 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:44.054272 5088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:44.054560 5088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:44.057382 5088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/ts-2/data/instance:
uuid: "2fa7441dfd4a46c5834c4a1ca4d18b20"
format_stamp: "Formatted at 2026-08-11 22:25:44 on dist-test-slave-gjrz"
server_key: "7939a1adb1cc9036b78a848cf7e4a132"
server_key_iv: "be85376bea5ba4472da4377996580d6a"
server_key_version: "encryptionkey@0"
I20260811 22:25:44.058135 5088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance:
uuid: "2fa7441dfd4a46c5834c4a1ca4d18b20"
format_stamp: "Formatted at 2026-08-11 22:25:44 on dist-test-slave-gjrz"
server_key: "7939a1adb1cc9036b78a848cf7e4a132"
server_key_iv: "be85376bea5ba4472da4377996580d6a"
server_key_version: "encryptionkey@0"
I20260811 22:25:44.062952 5088 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260811 22:25:44.066279 5116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:44.067600 5088 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260811 22:25:44.067745 5088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/ts-2/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/ts-2/wal
uuid: "2fa7441dfd4a46c5834c4a1ca4d18b20"
format_stamp: "Formatted at 2026-08-11 22:25:44 on dist-test-slave-gjrz"
server_key: "7939a1adb1cc9036b78a848cf7e4a132"
server_key_iv: "be85376bea5ba4472da4377996580d6a"
server_key_version: "encryptionkey@0"
I20260811 22:25:44.067920 5088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:44.085024 5088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:44.089248 5088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:44.089512 5088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:44.090394 5088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:25:44.091694 5088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:25:44.091768 5088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:44.091899 5088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:25:44.091948 5088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:44.120695 5088 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.3:38471
I20260811 22:25:44.120733 5311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.3:38471 every 8 connection(s)
I20260811 22:25:44.121909 5088 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
I20260811 22:25:44.128707 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 5088
I20260811 22:25:44.128860 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance
I20260811 22:25:44.129406 30120 external_mini_cluster.cc:1428] Setting key 53138b879be6ba1c9da0aea6ddce8b18
Aug 11 22:25:44 dist-test-slave-gjrz krb5kdc[4372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487144, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.3@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:25:44.141696 4824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:44.124359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:46459 (local address 127.29.106.62:38813)
0811 22:25:44.124577 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:44.124584 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:44.125702 (+ 1118us) server_negotiation.cc:407] Connection header received
0811 22:25:44.127474 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:44.127478 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:44.127546 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:44.127635 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:44.129113 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:44.130100 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:44.131904 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:44.132458 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:44.132576 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:44.135948 (+ 3372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:44.135955 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:44.135960 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:44.136026 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:44.138619 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:44.139386 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:44.139390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:44.139392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:44.139523 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:44.139933 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:44.139937 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:44.139939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:44.140240 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:44.140396 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:44.141090 (+ 694us) server_negotiation.cc:299] Negotiation successful
0811 22:25:44.141334 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260811 22:25:44.142616 5315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:44.124821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:38813 (local address 127.29.106.3:46459)
0811 22:25:44.125547 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:44.125586 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:44.127196 (+ 1610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:44.127821 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:44.127828 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:44.128344 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:44.128937 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:44.128961 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:44.130252 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:44.130258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:44.131761 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:44.131775 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:44.132691 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:44.132699 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:44.132922 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:44.133536 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:44.133559 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:44.135737 (+ 2178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:44.138773 (+ 3036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:44.138780 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:44.138792 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:44.139227 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:44.139610 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:44.139613 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:44.139615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:44.139803 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:44.140370 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:44.140379 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:44.140716 (+ 337us) client_negotiation.cc:769] Sending connection context
0811 22:25:44.140946 (+ 230us) client_negotiation.cc:240] Negotiation successful
0811 22:25:44.141452 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":278,"threads_started":1}
I20260811 22:25:44.144098 5312 heartbeater.cc:347] Connected to a master server at 127.29.106.62:38813
I20260811 22:25:44.144376 5312 heartbeater.cc:464] Registering TS with master...
I20260811 22:25:44.145186 5312 heartbeater.cc:511] Master 127.29.106.62:38813 requested a full tablet report, sending...
I20260811 22:25:44.146577 4453 ts_manager.cc:194] Registered new tserver with Master: 2fa7441dfd4a46c5834c4a1ca4d18b20 (127.29.106.3:38471)
I20260811 22:25:44.147594 4453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.3@KRBTEST.COM'} at 127.29.106.3:46459
I20260811 22:25:44.157212 30120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 22:25:44.185745 4821 heartbeater.cc:503] Master 127.29.106.62:38813 was elected leader, sending a full tablet report...
Aug 11 22:25:44 dist-test-slave-gjrz krb5kdc[4372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 11 22:25:44 dist-test-slave-gjrz krb5kdc[4372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.1@KRBTEST.COM
I20260811 22:25:44.208163 5327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:44.193667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38684 (local address 127.29.106.1:36455)
0811 22:25:44.194302 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:44.194309 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:44.194336 (+ 27us) server_negotiation.cc:407] Connection header received
0811 22:25:44.194393 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:44.194399 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:44.194607 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:44.194731 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:44.196091 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:44.197140 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:44.198389 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:44.198950 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:44.199079 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:44.202753 (+ 3674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:44.202765 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:44.202782 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:44.202821 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:44.205537 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:44.206140 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:44.206146 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:44.206148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:44.206272 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:44.206697 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:44.206701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:44.206703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:44.207060 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:44.207338 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:44.207647 (+ 309us) server_negotiation.cc:299] Negotiation successful
0811 22:25:44.207843 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":339,"threads_started":1}
Aug 11 22:25:44 dist-test-slave-gjrz krb5kdc[4372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:25:44.224609 4824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:44.212321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57566 (local address 127.29.106.62:38813)
0811 22:25:44.212519 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:44.212523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:44.212587 (+ 64us) server_negotiation.cc:407] Connection header received
0811 22:25:44.212725 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:44.212729 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:44.212790 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:44.212858 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:44.214001 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:44.214716 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:44.215909 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:44.216337 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:44.216426 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:44.219875 (+ 3449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:44.219880 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:44.219883 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:44.219915 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:44.222371 (+ 2456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:44.222897 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:44.222900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:44.222903 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:44.223029 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:44.223349 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:44.223352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:44.223354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:44.223665 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:44.223855 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:44.224076 (+ 221us) server_negotiation.cc:299] Negotiation successful
0811 22:25:44.224236 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260811 22:25:44.225243 4453 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57566
I20260811 22:25:44.226177 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 4566
I20260811 22:25:44.238386 5327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:44.227341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38690 (local address 127.29.106.1:36455)
0811 22:25:44.227527 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:44.227534 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:44.227681 (+ 147us) server_negotiation.cc:407] Connection header received
0811 22:25:44.227831 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:44.227837 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:44.227903 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:44.227981 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:44.229101 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:44.229750 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:44.230906 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:44.231394 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:44.231499 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:44.233530 (+ 2031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:44.233535 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:44.233538 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:44.233570 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:44.236001 (+ 2431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:44.236657 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:44.236663 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:44.236666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:44.236787 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:44.237103 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:44.237109 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:44.237112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:44.237349 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:44.237578 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:44.237880 (+ 302us) server_negotiation.cc:299] Negotiation successful
0811 22:25:44.238096 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260811 22:25:44.239092 4807 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:25:44.320123 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 4566
I20260811 22:25:44.346521 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 4837
Aug 11 22:25:44 dist-test-slave-gjrz krb5kdc[4372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.2@KRBTEST.COM
I20260811 22:25:44.361826 5337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:44.347424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49110 (local address 127.29.106.2:33107)
0811 22:25:44.348012 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:44.348019 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:44.348047 (+ 28us) server_negotiation.cc:407] Connection header received
0811 22:25:44.348116 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:44.348122 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:44.348336 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:44.348461 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:44.349603 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:44.350684 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:44.351996 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:44.352583 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:44.352680 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:44.356570 (+ 3890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:44.356581 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:44.356596 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:44.356628 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:44.359527 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:44.360122 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:44.360129 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:44.360131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:44.360245 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:44.360567 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:44.360570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:44.360572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:44.360886 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:44.361112 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:44.361320 (+ 208us) server_negotiation.cc:299] Negotiation successful
0811 22:25:44.361511 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":302,"threads_started":1}
I20260811 22:25:44.363145 5064 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:25:44.444871 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 4837
I20260811 22:25:44.469620 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 5088
Aug 11 22:25:44 dist-test-slave-gjrz krb5kdc[4372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.3@KRBTEST.COM
I20260811 22:25:44.485734 5339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:44.470630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55160 (local address 127.29.106.3:38471)
0811 22:25:44.471362 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:44.471369 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:44.471399 (+ 30us) server_negotiation.cc:407] Connection header received
0811 22:25:44.471464 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:44.471470 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:44.471679 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:44.471804 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:44.473093 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:44.474164 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:44.475460 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:44.476027 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:44.476132 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:44.480181 (+ 4049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:44.480193 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:44.480207 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:44.480241 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:44.482928 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:44.483691 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:44.483699 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:44.483701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:44.483841 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:44.484210 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:44.484213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:44.484215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:44.484552 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:44.484845 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:44.485164 (+ 319us) server_negotiation.cc:299] Negotiation successful
0811 22:25:44.485420 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":393,"threads_started":1}
I20260811 22:25:44.486976 5303 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:25:44.568449 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 5088
I20260811 22:25:44.592926 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 4395
I20260811 22:25:44.594067 4509 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:25:44.696959 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 4395
2026-08-11T22:25:44Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4176 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:25:44 dist-test-slave-gjrz krb5kdc[5347](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:25:44 dist-test-slave-gjrz krb5kdc[5347](info): set up 2 sockets
Aug 11 22:25:44 dist-test-slave-gjrz krb5kdc[5347](info): commencing operation
krb5kdc: starting...
W20260811 22:25:45.941494 30120 mini_kdc.cc:121] Time spent starting KDC: real 1.191s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Aug 11 22:25:46 dist-test-slave-gjrz krb5kdc[5347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:25:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:25:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.106.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
I20260811 22:25:46.134382 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:41209
--webserver_interface=127.29.106.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:36453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:41209
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:25:46.454207 5375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:46.454464 5375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:46.454607 5375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:46.461514 5375 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:25:46.461627 5375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:46.461706 5375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:46.461786 5375 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:25:46.461848 5375 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:25:46.471117 5375 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:36453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:41209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:41209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.106.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:25:46.472244 5375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:46.473471 5375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:46.484067 5381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:46.484156 5383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:46.484179 5386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:46.485033 5375 server_base.cc:1034] running on GCE node
I20260811 22:25:46.485706 5375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:46.486953 5375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:46.488161 5375 hybrid_clock.cc:648] HybridClock initialized: now 1786487146488106 us; error 40 us; skew 500 ppm
I20260811 22:25:46.488494 5375 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:25:46 dist-test-slave-gjrz krb5kdc[5347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487146, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:46.491670 5375 init.cc:377] Logged in from keytab as kudu/127.29.106.62@KRBTEST.COM (short username kudu)
I20260811 22:25:46.493021 5375 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:46.493134 5375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:46.493499 5375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:46.497340 5375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "a6a63a5163044443b0301e1ac8172a20"
format_stamp: "Formatted at 2026-08-11 22:25:46 on dist-test-slave-gjrz"
server_key: "e4a73544edee202156c031c570c8120a"
server_key_iv: "951cd0e9e95dcf088a39839d36527672"
server_key_version: "encryptionkey@0"
I20260811 22:25:46.498322 5375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "a6a63a5163044443b0301e1ac8172a20"
format_stamp: "Formatted at 2026-08-11 22:25:46 on dist-test-slave-gjrz"
server_key: "e4a73544edee202156c031c570c8120a"
server_key_iv: "951cd0e9e95dcf088a39839d36527672"
server_key_version: "encryptionkey@0"
I20260811 22:25:46.503736 5375 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260811 22:25:46.508275 5398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:46.509838 5375 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 22:25:46.510011 5375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "a6a63a5163044443b0301e1ac8172a20"
format_stamp: "Formatted at 2026-08-11 22:25:46 on dist-test-slave-gjrz"
server_key: "e4a73544edee202156c031c570c8120a"
server_key_iv: "951cd0e9e95dcf088a39839d36527672"
server_key_version: "encryptionkey@0"
I20260811 22:25:46.510228 5375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:46.543661 5375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:46.547691 5375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:46.547927 5375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:46.564719 5375 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.62:41209
I20260811 22:25:46.564746 5499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.62:41209 every 8 connection(s)
I20260811 22:25:46.565548 5375 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:25:46.568994 5500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:46.572865 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 5375
I20260811 22:25:46.573077 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:25:46.573850 30120 external_mini_cluster.cc:1428] Setting key ce8d1f6ec7c40a0b7cea1bef5ae23820
I20260811 22:25:46.580366 5500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6a63a5163044443b0301e1ac8172a20: Bootstrap starting.
I20260811 22:25:46.583231 5500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6a63a5163044443b0301e1ac8172a20: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:46.584220 5500 log.cc:824] T 00000000000000000000000000000000 P a6a63a5163044443b0301e1ac8172a20: Log is configured to *not* fsync() on all Append() calls
I20260811 22:25:46.586596 5500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6a63a5163044443b0301e1ac8172a20: No bootstrap required, opened a new log
Aug 11 22:25:46 dist-test-slave-gjrz krb5kdc[5347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:25:46.594352 5500 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6a63a5163044443b0301e1ac8172a20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a63a5163044443b0301e1ac8172a20" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 41209 } }
I20260811 22:25:46.594702 5500 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6a63a5163044443b0301e1ac8172a20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:25:46.594802 5500 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6a63a5163044443b0301e1ac8172a20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6a63a5163044443b0301e1ac8172a20, State: Initialized, Role: FOLLOWER
I20260811 22:25:46.595095 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:46.575895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47256 (local address 127.29.106.62:41209)
0811 22:25:46.576628 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:46.576639 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:46.576679 (+ 40us) server_negotiation.cc:407] Connection header received
0811 22:25:46.578779 (+ 2100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:46.578801 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:46.579175 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:46.579607 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:46.581128 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:46.582463 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:46.583729 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:46.584390 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:46.584553 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:46.588369 (+ 3816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:46.588377 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:46.588390 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:46.588436 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:46.591300 (+ 2864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:46.591929 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:46.591937 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:46.591943 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:46.592112 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:46.592478 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:46.592487 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:46.592490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:46.593013 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:46.593314 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:46.593721 (+ 407us) server_negotiation.cc:299] Negotiation successful
0811 22:25:46.594091 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":352,"threads_started":1}
I20260811 22:25:46.595429 5500 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6a63a5163044443b0301e1ac8172a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a63a5163044443b0301e1ac8172a20" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 41209 } }
I20260811 22:25:46.595670 5500 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6a63a5163044443b0301e1ac8172a20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:25:46.595813 5500 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6a63a5163044443b0301e1ac8172a20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:25:46.595969 5500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6a63a5163044443b0301e1ac8172a20 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:46.598286 5500 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6a63a5163044443b0301e1ac8172a20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a63a5163044443b0301e1ac8172a20" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 41209 } }
I20260811 22:25:46.598733 5500 leader_election.cc:302] T 00000000000000000000000000000000 P a6a63a5163044443b0301e1ac8172a20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6a63a5163044443b0301e1ac8172a20; no voters:
I20260811 22:25:46.599231 5500 leader_election.cc:288] T 00000000000000000000000000000000 P a6a63a5163044443b0301e1ac8172a20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:25:46.599390 5509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6a63a5163044443b0301e1ac8172a20 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:25:46.600013 5509 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6a63a5163044443b0301e1ac8172a20 [term 1 LEADER]: Becoming Leader. State: Replica: a6a63a5163044443b0301e1ac8172a20, State: Running, Role: LEADER
I20260811 22:25:46.600488 5509 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6a63a5163044443b0301e1ac8172a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a63a5163044443b0301e1ac8172a20" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 41209 } }
I20260811 22:25:46.601148 5500 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6a63a5163044443b0301e1ac8172a20 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:25:46.603072 5511 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6a63a5163044443b0301e1ac8172a20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6a63a5163044443b0301e1ac8172a20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a63a5163044443b0301e1ac8172a20" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 41209 } } }
I20260811 22:25:46.603489 5511 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6a63a5163044443b0301e1ac8172a20 [sys.catalog]: This master's current role is: LEADER
I20260811 22:25:46.604048 5512 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6a63a5163044443b0301e1ac8172a20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6a63a5163044443b0301e1ac8172a20. Latest consensus state: current_term: 1 leader_uuid: "a6a63a5163044443b0301e1ac8172a20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a63a5163044443b0301e1ac8172a20" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 41209 } } }
I20260811 22:25:46.604354 5512 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6a63a5163044443b0301e1ac8172a20 [sys.catalog]: This master's current role is: LEADER
I20260811 22:25:46.609467 5522 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:25:46.615231 5522 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:25:46.624369 5522 catalog_manager.cc:1347] Generated new cluster ID: f0c08e3b06b14ef68b61c2272187f28a
I20260811 22:25:46.624478 5522 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:25:46.638237 5522 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:25:46.639498 5522 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:25:46.646279 5522 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6a63a5163044443b0301e1ac8172a20: Generated new TSK 0
I20260811 22:25:46.646960 5522 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.106.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab.
Entry for principal kudu/127.29.106.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab.
I20260811 22:25:46.706130 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.1:0
--local_ip_for_outbound_sockets=127.29.106.1
--webserver_interface=127.29.106.1
--webserver_port=0
--tserver_master_addrs=127.29.106.62:41209
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab
--principal=kudu/127.29.106.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:36453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:25:47.025738 5539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:47.025978 5539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:47.026090 5539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:47.032869 5539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:47.032959 5539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:47.033082 5539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.1
I20260811 22:25:47.042265 5539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:36453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab
--principal=kudu/127.29.106.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.106.1
--webserver_port=0
--tserver_master_addrs=127.29.106.62:41209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.1
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:25:47.043330 5539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:47.044490 5539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:47.056528 5547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:47.056557 5549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:47.056460 5554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:47.057346 5539 server_base.cc:1034] running on GCE node
I20260811 22:25:47.057868 5539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:47.058666 5539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:47.059850 5539 hybrid_clock.cc:648] HybridClock initialized: now 1786487147059810 us; error 32 us; skew 500 ppm
I20260811 22:25:47.060120 5539 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:25:47 dist-test-slave-gjrz krb5kdc[5347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487147, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:47.062990 5539 init.cc:377] Logged in from keytab as kudu/127.29.106.1@KRBTEST.COM (short username kudu)
I20260811 22:25:47.064038 5539 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:47.064148 5539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:47.064433 5539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:47.067301 5539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-0/data/instance:
uuid: "baa87368342f48f3a13217d00339ea1f"
format_stamp: "Formatted at 2026-08-11 22:25:47 on dist-test-slave-gjrz"
server_key: "c07b34812e8cf52955d67043253e57ef"
server_key_iv: "c90a530789d9e017ec0c76f34ae97f39"
server_key_version: "encryptionkey@0"
I20260811 22:25:47.068104 5539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance:
uuid: "baa87368342f48f3a13217d00339ea1f"
format_stamp: "Formatted at 2026-08-11 22:25:47 on dist-test-slave-gjrz"
server_key: "c07b34812e8cf52955d67043253e57ef"
server_key_iv: "c90a530789d9e017ec0c76f34ae97f39"
server_key_version: "encryptionkey@0"
I20260811 22:25:47.072716 5539 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 22:25:47.075891 5571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:47.077080 5539 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:25:47.077232 5539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-0/wal
uuid: "baa87368342f48f3a13217d00339ea1f"
format_stamp: "Formatted at 2026-08-11 22:25:47 on dist-test-slave-gjrz"
server_key: "c07b34812e8cf52955d67043253e57ef"
server_key_iv: "c90a530789d9e017ec0c76f34ae97f39"
server_key_version: "encryptionkey@0"
I20260811 22:25:47.077414 5539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:47.094357 5539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:47.098174 5539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:47.098404 5539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:47.099255 5539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:25:47.100500 5539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:25:47.100600 5539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:47.100709 5539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:25:47.100780 5539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:47.130522 5539 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.1:40963
I20260811 22:25:47.130577 5768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.1:40963 every 8 connection(s)
I20260811 22:25:47.131345 5539 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
I20260811 22:25:47.136982 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 5539
I20260811 22:25:47.137173 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance
I20260811 22:25:47.137794 30120 external_mini_cluster.cc:1428] Setting key ea511eab04a6df037ffc5a690f147dc5
Aug 11 22:25:47 dist-test-slave-gjrz krb5kdc[5347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487147, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.1@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:25:47.150877 5776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:47.133688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:40655 (local address 127.29.106.62:41209)
0811 22:25:47.134103 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:47.134107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:47.135025 (+ 918us) server_negotiation.cc:407] Connection header received
0811 22:25:47.136845 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:47.136851 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:47.136932 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:47.137037 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:47.138600 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:47.139518 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:47.141124 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:47.141685 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:47.141811 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:47.144988 (+ 3177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:47.144995 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:47.144999 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:47.145038 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:47.147905 (+ 2867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:47.148790 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:47.148795 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:47.148799 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:47.148932 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:47.149343 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:47.149346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:47.149348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:47.149617 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:47.149728 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:47.150468 (+ 740us) server_negotiation.cc:299] Negotiation successful
0811 22:25:47.150635 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":193,"threads_started":1}
I20260811 22:25:47.152144 5777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:47.134206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:41209 (local address 127.29.106.1:40655)
0811 22:25:47.134872 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:47.134915 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:47.136588 (+ 1673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:47.137282 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:47.137293 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:47.137800 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:47.138402 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:47.138434 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:47.139666 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:47.139671 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:47.140908 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:47.140954 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:47.141885 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:47.141890 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:47.142086 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:47.142683 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:47.142704 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:47.144766 (+ 2062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:47.148037 (+ 3271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:47.148046 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:47.148062 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:47.148676 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:47.149032 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:47.149038 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:47.149041 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:47.149256 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:47.149734 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:47.149741 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:47.150119 (+ 378us) client_negotiation.cc:769] Sending connection context
0811 22:25:47.150398 (+ 279us) client_negotiation.cc:240] Negotiation successful
0811 22:25:47.150917 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":267,"threads_started":1}
I20260811 22:25:47.153538 5771 heartbeater.cc:347] Connected to a master server at 127.29.106.62:41209
I20260811 22:25:47.153821 5771 heartbeater.cc:464] Registering TS with master...
I20260811 22:25:47.154425 5771 heartbeater.cc:511] Master 127.29.106.62:41209 requested a full tablet report, sending...
I20260811 22:25:47.156423 5430 ts_manager.cc:194] Registered new tserver with Master: baa87368342f48f3a13217d00339ea1f (127.29.106.1:40963)
I20260811 22:25:47.157799 5430 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.1@KRBTEST.COM'} at 127.29.106.1:40655
WARNING: no policy specified for kudu/127.29.106.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab.
Entry for principal kudu/127.29.106.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab.
I20260811 22:25:47.185832 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.2:0
--local_ip_for_outbound_sockets=127.29.106.2
--webserver_interface=127.29.106.2
--webserver_port=0
--tserver_master_addrs=127.29.106.62:41209
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab
--principal=kudu/127.29.106.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:36453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:25:47.498122 5783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:47.498395 5783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:47.498541 5783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:47.505627 5783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:47.505733 5783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:47.505877 5783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.2
I20260811 22:25:47.515221 5783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:36453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab
--principal=kudu/127.29.106.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.106.2
--webserver_port=0
--tserver_master_addrs=127.29.106.62:41209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.2
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:25:47.516402 5783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:47.517601 5783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:47.529225 5793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:47.529356 5791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:47.529444 5803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:47.530231 5783 server_base.cc:1034] running on GCE node
I20260811 22:25:47.530794 5783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:47.531713 5783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:47.532896 5783 hybrid_clock.cc:648] HybridClock initialized: now 1786487147532836 us; error 51 us; skew 500 ppm
I20260811 22:25:47.533174 5783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:25:47 dist-test-slave-gjrz krb5kdc[5347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487147, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:47.536026 5783 init.cc:377] Logged in from keytab as kudu/127.29.106.2@KRBTEST.COM (short username kudu)
I20260811 22:25:47.537020 5783 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:47.537104 5783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:47.537340 5783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:47.540042 5783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-1/data/instance:
uuid: "d71d6d24b717473e80e9cd393d0b8f6c"
format_stamp: "Formatted at 2026-08-11 22:25:47 on dist-test-slave-gjrz"
server_key: "243b8273d51bf658b88b8be397d582e9"
server_key_iv: "3fb70d9308e70343fb544152d31ac24f"
server_key_version: "encryptionkey@0"
I20260811 22:25:47.540781 5783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance:
uuid: "d71d6d24b717473e80e9cd393d0b8f6c"
format_stamp: "Formatted at 2026-08-11 22:25:47 on dist-test-slave-gjrz"
server_key: "243b8273d51bf658b88b8be397d582e9"
server_key_iv: "3fb70d9308e70343fb544152d31ac24f"
server_key_version: "encryptionkey@0"
I20260811 22:25:47.545444 5783 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 22:25:47.548584 5817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:47.549772 5783 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260811 22:25:47.549916 5783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-1/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-1/wal
uuid: "d71d6d24b717473e80e9cd393d0b8f6c"
format_stamp: "Formatted at 2026-08-11 22:25:47 on dist-test-slave-gjrz"
server_key: "243b8273d51bf658b88b8be397d582e9"
server_key_iv: "3fb70d9308e70343fb544152d31ac24f"
server_key_version: "encryptionkey@0"
I20260811 22:25:47.550087 5783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:47.584702 5783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:47.588833 5783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:47.589061 5783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:47.589956 5783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:25:47.591219 5783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:25:47.591337 5783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:47.591437 5783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:25:47.591498 5783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:47.620349 5783 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.2:40065
I20260811 22:25:47.620409 6039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.2:40065 every 8 connection(s)
I20260811 22:25:47.621119 5783 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
I20260811 22:25:47.623011 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 5783
I20260811 22:25:47.623168 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance
I20260811 22:25:47.623735 30120 external_mini_cluster.cc:1428] Setting key 0e11a859ff31dc7292a1a1c9bdffa8c3
Aug 11 22:25:47 dist-test-slave-gjrz krb5kdc[5347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487147, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.2@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:25:47.641010 5776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:47.623540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:57999 (local address 127.29.106.62:41209)
0811 22:25:47.623740 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:47.623747 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:47.625025 (+ 1278us) server_negotiation.cc:407] Connection header received
0811 22:25:47.626865 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:47.626871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:47.626952 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:47.627041 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:47.628525 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:47.629424 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:47.630976 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:47.631447 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:47.631561 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:47.635042 (+ 3481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:47.635049 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:47.635053 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:47.635092 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:47.638053 (+ 2961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:47.638739 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:47.638744 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:47.638747 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:47.638907 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:47.639318 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:47.639323 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:47.639327 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:47.639680 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:47.639831 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:47.640445 (+ 614us) server_negotiation.cc:299] Negotiation successful
0811 22:25:47.640682 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260811 22:25:47.641640 6047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:47.624062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:41209 (local address 127.29.106.2:57999)
0811 22:25:47.624871 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:47.624918 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:47.626618 (+ 1700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:47.627306 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:47.627319 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:47.627817 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:47.628358 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:47.628375 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:47.629602 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:47.629606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:47.630830 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:47.630844 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:47.631666 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:47.631674 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:47.631883 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:47.632491 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:47.632512 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:47.634803 (+ 2291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:47.638190 (+ 3387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:47.638196 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:47.638207 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:47.638615 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:47.639004 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:47.639007 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:47.639009 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:47.639180 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:47.639795 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:47.639800 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:47.640088 (+ 288us) client_negotiation.cc:769] Sending connection context
0811 22:25:47.640310 (+ 222us) client_negotiation.cc:240] Negotiation successful
0811 22:25:47.640651 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":322,"threads_started":1}
I20260811 22:25:47.643149 6045 heartbeater.cc:347] Connected to a master server at 127.29.106.62:41209
I20260811 22:25:47.643455 6045 heartbeater.cc:464] Registering TS with master...
I20260811 22:25:47.644057 6045 heartbeater.cc:511] Master 127.29.106.62:41209 requested a full tablet report, sending...
I20260811 22:25:47.645488 5430 ts_manager.cc:194] Registered new tserver with Master: d71d6d24b717473e80e9cd393d0b8f6c (127.29.106.2:40065)
WARNING: no policy specified for kudu/127.29.106.3@KRBTEST.COM; defaulting to no policy
I20260811 22:25:47.646509 5430 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.2@KRBTEST.COM'} at 127.29.106.2:57999
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab.
Entry for principal kudu/127.29.106.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab.
I20260811 22:25:47.669813 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.3:0
--local_ip_for_outbound_sockets=127.29.106.3
--webserver_interface=127.29.106.3
--webserver_port=0
--tserver_master_addrs=127.29.106.62:41209
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab
--principal=kudu/127.29.106.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:36453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:25:47.982488 6050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:47.982735 6050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:47.982831 6050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:47.989813 6050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:47.989908 6050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:47.990029 6050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.3
I20260811 22:25:47.999138 6050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:36453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab
--principal=kudu/127.29.106.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.106.3
--webserver_port=0
--tserver_master_addrs=127.29.106.62:41209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.3
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:25:48.000262 6050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:48.001466 6050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:48.013579 6068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:48.013516 6074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:48.013557 6069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:48.014470 6050 server_base.cc:1034] running on GCE node
I20260811 22:25:48.015069 6050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:48.016019 6050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:48.017210 6050 hybrid_clock.cc:648] HybridClock initialized: now 1786487148017158 us; error 44 us; skew 500 ppm
I20260811 22:25:48.017484 6050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:25:48 dist-test-slave-gjrz krb5kdc[5347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487148, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:48.020401 6050 init.cc:377] Logged in from keytab as kudu/127.29.106.3@KRBTEST.COM (short username kudu)
I20260811 22:25:48.021512 6050 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:48.021602 6050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:48.021914 6050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:48.024729 6050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-2/data/instance:
uuid: "04acd70546f74aabb52e11f0400d733a"
format_stamp: "Formatted at 2026-08-11 22:25:48 on dist-test-slave-gjrz"
server_key: "9aec4592b7a143643784c641e473ddd0"
server_key_iv: "15515fabe95aeccf0b4782ab18a5f217"
server_key_version: "encryptionkey@0"
I20260811 22:25:48.025566 6050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance:
uuid: "04acd70546f74aabb52e11f0400d733a"
format_stamp: "Formatted at 2026-08-11 22:25:48 on dist-test-slave-gjrz"
server_key: "9aec4592b7a143643784c641e473ddd0"
server_key_iv: "15515fabe95aeccf0b4782ab18a5f217"
server_key_version: "encryptionkey@0"
I20260811 22:25:48.030305 6050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260811 22:25:48.033547 6085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:48.034737 6050 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260811 22:25:48.034879 6050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-2/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-2/wal
uuid: "04acd70546f74aabb52e11f0400d733a"
format_stamp: "Formatted at 2026-08-11 22:25:48 on dist-test-slave-gjrz"
server_key: "9aec4592b7a143643784c641e473ddd0"
server_key_iv: "15515fabe95aeccf0b4782ab18a5f217"
server_key_version: "encryptionkey@0"
I20260811 22:25:48.035099 6050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:48.048405 6050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:48.052327 6050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:48.052585 6050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:48.053452 6050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:25:48.054682 6050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:25:48.054769 6050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:48.054867 6050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:25:48.054941 6050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:48.083401 6050 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.3:37673
I20260811 22:25:48.083446 6300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.3:37673 every 8 connection(s)
I20260811 22:25:48.084152 6050 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
I20260811 22:25:48.086551 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 6050
I20260811 22:25:48.086705 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance
I20260811 22:25:48.087110 30120 external_mini_cluster.cc:1428] Setting key b0c66fb89d8b694e1daeec6bce59f7fa
Aug 11 22:25:48 dist-test-slave-gjrz krb5kdc[5347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487148, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.3@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:25:48.102519 5776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:48.086342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:60051 (local address 127.29.106.62:41209)
0811 22:25:48.086558 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:48.086565 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:48.087732 (+ 1167us) server_negotiation.cc:407] Connection header received
0811 22:25:48.089678 (+ 1946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:48.089684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:48.089763 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:48.089846 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:48.091378 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:48.092119 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:48.093604 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:48.094017 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:48.094136 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:48.097216 (+ 3080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:48.097221 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:48.097224 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:48.097256 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:48.099585 (+ 2329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:48.100263 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:48.100266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:48.100269 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:48.100415 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:48.100913 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:48.100918 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:48.100921 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:48.101236 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:48.101378 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:48.102123 (+ 745us) server_negotiation.cc:299] Negotiation successful
0811 22:25:48.102280 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260811 22:25:48.103420 6307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:48.086801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:41209 (local address 127.29.106.3:60051)
0811 22:25:48.087576 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:48.087621 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:48.089417 (+ 1796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:48.090200 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:48.090208 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:48.090683 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:48.091198 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:48.091215 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:48.092265 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:48.092269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:48.093462 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:48.093475 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:48.094250 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:48.094258 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:48.094455 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:48.094996 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:48.095021 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:48.097018 (+ 1997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:48.099708 (+ 2690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:48.099716 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:48.099730 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:48.100142 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:48.100534 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:48.100538 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:48.100542 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:48.100770 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:48.101393 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:48.101401 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:48.101737 (+ 336us) client_negotiation.cc:769] Sending connection context
0811 22:25:48.102037 (+ 300us) client_negotiation.cc:240] Negotiation successful
0811 22:25:48.102412 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":303,"threads_started":1}
I20260811 22:25:48.104573 6304 heartbeater.cc:347] Connected to a master server at 127.29.106.62:41209
I20260811 22:25:48.104830 6304 heartbeater.cc:464] Registering TS with master...
I20260811 22:25:48.105406 6304 heartbeater.cc:511] Master 127.29.106.62:41209 requested a full tablet report, sending...
I20260811 22:25:48.106671 5430 ts_manager.cc:194] Registered new tserver with Master: 04acd70546f74aabb52e11f0400d733a (127.29.106.3:37673)
I20260811 22:25:48.107486 5430 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.3@KRBTEST.COM'} at 127.29.106.3:60051
I20260811 22:25:48.113761 30120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 22:25:48.129798 5776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:48.118112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47264 (local address 127.29.106.62:41209)
0811 22:25:48.118264 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:48.118267 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:48.118778 (+ 511us) server_negotiation.cc:407] Connection header received
0811 22:25:48.118928 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:48.118933 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:48.118997 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:48.119093 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:48.120462 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:48.121160 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:48.122404 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:48.122844 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:48.122951 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:48.125037 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:48.125049 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:48.125052 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:48.125082 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:48.127452 (+ 2370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:48.128030 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:48.128034 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:48.128036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:48.128166 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:48.128505 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:48.128509 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:48.128511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:48.128793 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:48.129013 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:48.129324 (+ 311us) server_negotiation.cc:299] Negotiation successful
0811 22:25:48.129522 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260811 22:25:48.133750 5430 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47264:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:25:48.141603 5430 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:25:48.160652 5771 heartbeater.cc:503] Master 127.29.106.62:41209 was elected leader, sending a full tablet report...
I20260811 22:25:48.166993 6323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:48.153182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.3:37673 (local address 127.0.0.1:57966)
0811 22:25:48.155198 (+ 2016us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:48.155255 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:48.155460 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:48.157628 (+ 2168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:48.157636 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:48.157670 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:25:48.158067 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:48.158080 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:48.161308 (+ 3228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:48.161314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:48.163852 (+ 2538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:48.163871 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:48.166024 (+ 2153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:48.166032 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:48.166275 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:48.166331 (+ 56us) client_negotiation.cc:769] Sending connection context
0811 22:25:48.166464 (+ 133us) client_negotiation.cc:240] Negotiation successful
0811 22:25:48.166640 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1748,"thread_start_us":1589,"threads_started":1}
I20260811 22:25:48.167199 6325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:48.153280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57966 (local address 127.29.106.3:37673)
0811 22:25:48.153922 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:48.153931 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:48.155651 (+ 1720us) server_negotiation.cc:407] Connection header received
0811 22:25:48.155763 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:48.155771 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:48.156028 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:48.156218 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:25:48.158345 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:48.161167 (+ 2822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:48.164066 (+ 2899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:48.165817 (+ 1751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:48.165943 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:48.166228 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:48.166724 (+ 496us) server_negotiation.cc:299] Negotiation successful
0811 22:25:48.166895 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":1183,"threads_started":1}
I20260811 22:25:48.171383 6176 tablet_service.cc:1467] Processing CreateTablet for tablet 6c4c21ad7c974e3b885565d1d45c61b2 (DEFAULT_TABLE table=test-table [id=88f7f6ba704943cb95ad7534614df41a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:25:48.172634 6176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c4c21ad7c974e3b885565d1d45c61b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:48.174602 6329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:48.153801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:40963 (local address 127.0.0.1:55324)
0811 22:25:48.165146 (+ 11345us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:48.165166 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:48.165293 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:48.168254 (+ 2961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:48.168258 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:48.168283 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:25:48.168649 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:48.168664 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:48.170377 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:48.170405 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:48.172424 (+ 2019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:48.172440 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:48.173860 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:48.173866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:48.174022 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:48.174064 (+ 42us) client_negotiation.cc:769] Sending connection context
0811 22:25:48.174151 (+ 87us) client_negotiation.cc:240] Negotiation successful
0811 22:25:48.174275 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11207,"thread_start_us":11295,"threads_started":1}
I20260811 22:25:48.174692 6338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:48.154479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55324 (local address 127.29.106.1:40963)
0811 22:25:48.167819 (+ 13340us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:48.167824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:48.167847 (+ 23us) server_negotiation.cc:407] Connection header received
0811 22:25:48.167914 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:48.167918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:48.168083 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:48.168246 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:25:48.168838 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:48.170254 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:48.172622 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:48.173663 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:48.173787 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:48.174011 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:48.174207 (+ 196us) server_negotiation.cc:299] Negotiation successful
0811 22:25:48.174364 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13193,"thread_start_us":13059,"threads_started":1}
I20260811 22:25:48.175701 6339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:48.155653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.2:40065 (local address 127.0.0.1:54698)
0811 22:25:48.165740 (+ 10087us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:48.165758 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:48.165868 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:48.168575 (+ 2707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:48.168579 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:48.168613 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:25:48.169012 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:48.169025 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:48.170859 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:48.170865 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:48.172889 (+ 2024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:48.172903 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:48.174937 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:48.174943 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:48.175115 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:48.175153 (+ 38us) client_negotiation.cc:769] Sending connection context
0811 22:25:48.175243 (+ 90us) client_negotiation.cc:240] Negotiation successful
0811 22:25:48.175385 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9960,"thread_start_us":9821,"threads_started":1}
I20260811 22:25:48.176473 6340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:48.156179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54698 (local address 127.29.106.2:40065)
0811 22:25:48.166698 (+ 10519us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:48.166708 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:48.166740 (+ 32us) server_negotiation.cc:407] Connection header received
0811 22:25:48.166852 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:48.166861 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:48.167140 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:48.168589 (+ 1449us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:25:48.169243 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:48.170709 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:48.173058 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:48.174490 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:48.174626 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:48.174797 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:48.175993 (+ 1196us) server_negotiation.cc:299] Negotiation successful
0811 22:25:48.176165 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10340,"thread_start_us":10217,"threads_started":1}
I20260811 22:25:48.178537 5655 tablet_service.cc:1467] Processing CreateTablet for tablet 6c4c21ad7c974e3b885565d1d45c61b2 (DEFAULT_TABLE table=test-table [id=88f7f6ba704943cb95ad7534614df41a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:25:48.179728 5655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c4c21ad7c974e3b885565d1d45c61b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:48.180109 5912 tablet_service.cc:1467] Processing CreateTablet for tablet 6c4c21ad7c974e3b885565d1d45c61b2 (DEFAULT_TABLE table=test-table [id=88f7f6ba704943cb95ad7534614df41a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:25:48.181433 5912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c4c21ad7c974e3b885565d1d45c61b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:48.186851 6342 tablet_bootstrap.cc:492] T 6c4c21ad7c974e3b885565d1d45c61b2 P 04acd70546f74aabb52e11f0400d733a: Bootstrap starting.
I20260811 22:25:48.190455 6342 tablet_bootstrap.cc:654] T 6c4c21ad7c974e3b885565d1d45c61b2 P 04acd70546f74aabb52e11f0400d733a: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:48.191807 6342 log.cc:824] T 6c4c21ad7c974e3b885565d1d45c61b2 P 04acd70546f74aabb52e11f0400d733a: Log is configured to *not* fsync() on all Append() calls
I20260811 22:25:48.193929 6345 tablet_bootstrap.cc:492] T 6c4c21ad7c974e3b885565d1d45c61b2 P d71d6d24b717473e80e9cd393d0b8f6c: Bootstrap starting.
I20260811 22:25:48.194679 6342 tablet_bootstrap.cc:492] T 6c4c21ad7c974e3b885565d1d45c61b2 P 04acd70546f74aabb52e11f0400d733a: No bootstrap required, opened a new log
I20260811 22:25:48.194913 6342 ts_tablet_manager.cc:1397] T 6c4c21ad7c974e3b885565d1d45c61b2 P 04acd70546f74aabb52e11f0400d733a: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260811 22:25:48.195959 6344 tablet_bootstrap.cc:492] T 6c4c21ad7c974e3b885565d1d45c61b2 P baa87368342f48f3a13217d00339ea1f: Bootstrap starting.
I20260811 22:25:48.197077 6345 tablet_bootstrap.cc:654] T 6c4c21ad7c974e3b885565d1d45c61b2 P d71d6d24b717473e80e9cd393d0b8f6c: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:48.198297 6345 log.cc:824] T 6c4c21ad7c974e3b885565d1d45c61b2 P d71d6d24b717473e80e9cd393d0b8f6c: Log is configured to *not* fsync() on all Append() calls
I20260811 22:25:48.199184 6344 tablet_bootstrap.cc:654] T 6c4c21ad7c974e3b885565d1d45c61b2 P baa87368342f48f3a13217d00339ea1f: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:48.200275 6344 log.cc:824] T 6c4c21ad7c974e3b885565d1d45c61b2 P baa87368342f48f3a13217d00339ea1f: Log is configured to *not* fsync() on all Append() calls
I20260811 22:25:48.202190 6345 tablet_bootstrap.cc:492] T 6c4c21ad7c974e3b885565d1d45c61b2 P d71d6d24b717473e80e9cd393d0b8f6c: No bootstrap required, opened a new log
I20260811 22:25:48.202450 6345 ts_tablet_manager.cc:1397] T 6c4c21ad7c974e3b885565d1d45c61b2 P d71d6d24b717473e80e9cd393d0b8f6c: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260811 22:25:48.202785 6342 raft_consensus.cc:348] T 6c4c21ad7c974e3b885565d1d45c61b2 P 04acd70546f74aabb52e11f0400d733a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04acd70546f74aabb52e11f0400d733a" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 37673 } } peers { permanent_uuid: "d71d6d24b717473e80e9cd393d0b8f6c" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 40065 } } peers { permanent_uuid: "baa87368342f48f3a13217d00339ea1f" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 40963 } }
I20260811 22:25:48.203115 6342 raft_consensus.cc:374] T 6c4c21ad7c974e3b885565d1d45c61b2 P 04acd70546f74aabb52e11f0400d733a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:25:48.203212 6342 raft_consensus.cc:729] T 6c4c21ad7c974e3b885565d1d45c61b2 P 04acd70546f74aabb52e11f0400d733a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04acd70546f74aabb52e11f0400d733a, State: Initialized, Role: FOLLOWER
I20260811 22:25:48.203382 6344 tablet_bootstrap.cc:492] T 6c4c21ad7c974e3b885565d1d45c61b2 P baa87368342f48f3a13217d00339ea1f: No bootstrap required, opened a new log
I20260811 22:25:48.203649 6344 ts_tablet_manager.cc:1397] T 6c4c21ad7c974e3b885565d1d45c61b2 P baa87368342f48f3a13217d00339ea1f: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260811 22:25:48.203851 6342 consensus_queue.cc:260] T 6c4c21ad7c974e3b885565d1d45c61b2 P 04acd70546f74aabb52e11f0400d733a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04acd70546f74aabb52e11f0400d733a" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 37673 } } peers { permanent_uuid: "d71d6d24b717473e80e9cd393d0b8f6c" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 40065 } } peers { permanent_uuid: "baa87368342f48f3a13217d00339ea1f" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 40963 } }
I20260811 22:25:48.204879 6304 heartbeater.cc:503] Master 127.29.106.62:41209 was elected leader, sending a full tablet report...
I20260811 22:25:48.206254 6350 raft_consensus.cc:482] T 6c4c21ad7c974e3b885565d1d45c61b2 P 04acd70546f74aabb52e11f0400d733a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:25:48.206598 6350 raft_consensus.cc:504] T 6c4c21ad7c974e3b885565d1d45c61b2 P 04acd70546f74aabb52e11f0400d733a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04acd70546f74aabb52e11f0400d733a" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 37673 } } peers { permanent_uuid: "d71d6d24b717473e80e9cd393d0b8f6c" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 40065 } } peers { permanent_uuid: "baa87368342f48f3a13217d00339ea1f" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 40963 } }
I20260811 22:25:48.211532 6342 ts_tablet_manager.cc:1428] T 6c4c21ad7c974e3b885565d1d45c61b2 P 04acd70546f74aabb52e11f0400d733a: Time spent starting tablet: real 0.017s user 0.011s sys 0.000s
I20260811 22:25:48.213726 6345 raft_consensus.cc:348] T 6c4c21ad7c974e3b885565d1d45c61b2 P d71d6d24b717473e80e9cd393d0b8f6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04acd70546f74aabb52e11f0400d733a" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 37673 } } peers { permanent_uuid: "d71d6d24b717473e80e9cd393d0b8f6c" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 40065 } } peers { permanent_uuid: "baa87368342f48f3a13217d00339ea1f" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 40963 } }
I20260811 22:25:48.213726 6344 raft_consensus.cc:348] T 6c4c21ad7c974e3b885565d1d45c61b2 P baa87368342f48f3a13217d00339ea1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04acd70546f74aabb52e11f0400d733a" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 37673 } } peers { permanent_uuid: "d71d6d24b717473e80e9cd393d0b8f6c" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 40065 } } peers { permanent_uuid: "baa87368342f48f3a13217d00339ea1f" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 40963 } }
I20260811 22:25:48.214154 6344 raft_consensus.cc:374] T 6c4c21ad7c974e3b885565d1d45c61b2 P baa87368342f48f3a13217d00339ea1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:25:48.214211 6345 raft_consensus.cc:374] T 6c4c21ad7c974e3b885565d1d45c61b2 P d71d6d24b717473e80e9cd393d0b8f6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:25:48.214293 6344 raft_consensus.cc:729] T 6c4c21ad7c974e3b885565d1d45c61b2 P baa87368342f48f3a13217d00339ea1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baa87368342f48f3a13217d00339ea1f, State: Initialized, Role: FOLLOWER
I20260811 22:25:48.214347 6345 raft_consensus.cc:729] T 6c4c21ad7c974e3b885565d1d45c61b2 P d71d6d24b717473e80e9cd393d0b8f6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d71d6d24b717473e80e9cd393d0b8f6c, State: Initialized, Role: FOLLOWER
I20260811 22:25:48.215025 6344 consensus_queue.cc:260] T 6c4c21ad7c974e3b885565d1d45c61b2 P baa87368342f48f3a13217d00339ea1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04acd70546f74aabb52e11f0400d733a" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 37673 } } peers { permanent_uuid: "d71d6d24b717473e80e9cd393d0b8f6c" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 40065 } } peers { permanent_uuid: "baa87368342f48f3a13217d00339ea1f" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 40963 } }
I20260811 22:25:48.215025 6345 consensus_queue.cc:260] T 6c4c21ad7c974e3b885565d1d45c61b2 P d71d6d24b717473e80e9cd393d0b8f6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04acd70546f74aabb52e11f0400d733a" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 37673 } } peers { permanent_uuid: "d71d6d24b717473e80e9cd393d0b8f6c" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 40065 } } peers { permanent_uuid: "baa87368342f48f3a13217d00339ea1f" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 40963 } }
I20260811 22:25:48.218528 6307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:48.208921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.2:40065 (local address 127.29.106.3:39189)
0811 22:25:48.209402 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:48.209424 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:48.209542 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:48.210599 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:48.210604 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:48.210630 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:25:48.210804 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:48.210814 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:48.212403 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:48.212409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:48.214490 (+ 2081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:48.214507 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:48.217779 (+ 3272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:48.217786 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:48.218021 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:48.218056 (+ 35us) client_negotiation.cc:769] Sending connection context
0811 22:25:48.218157 (+ 101us) client_negotiation.cc:240] Negotiation successful
0811 22:25:48.218284 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332}
I20260811 22:25:48.219161 6340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:48.209892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:39189 (local address 127.29.106.2:40065)
0811 22:25:48.210120 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:48.210126 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:48.210145 (+ 19us) server_negotiation.cc:407] Connection header received
0811 22:25:48.210220 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:48.210226 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:48.210305 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:48.210397 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:25:48.210968 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:48.212207 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:48.216085 (+ 3878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:48.217450 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:48.217609 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:48.217684 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:48.218798 (+ 1114us) server_negotiation.cc:299] Negotiation successful
0811 22:25:48.218921 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260811 22:25:48.220088 5953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c4c21ad7c974e3b885565d1d45c61b2" candidate_uuid: "04acd70546f74aabb52e11f0400d733a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d71d6d24b717473e80e9cd393d0b8f6c" is_pre_election: true
I20260811 22:25:48.220577 5953 raft_consensus.cc:2454] T 6c4c21ad7c974e3b885565d1d45c61b2 P d71d6d24b717473e80e9cd393d0b8f6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04acd70546f74aabb52e11f0400d733a in term 0.
I20260811 22:25:48.220767 6350 leader_election.cc:288] T 6c4c21ad7c974e3b885565d1d45c61b2 P 04acd70546f74aabb52e11f0400d733a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d71d6d24b717473e80e9cd393d0b8f6c (127.29.106.2:40065), baa87368342f48f3a13217d00339ea1f (127.29.106.1:40963)
I20260811 22:25:48.221448 6094 leader_election.cc:302] T 6c4c21ad7c974e3b885565d1d45c61b2 P 04acd70546f74aabb52e11f0400d733a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04acd70546f74aabb52e11f0400d733a, d71d6d24b717473e80e9cd393d0b8f6c; no voters:
I20260811 22:25:48.221921 6350 raft_consensus.cc:2781] T 6c4c21ad7c974e3b885565d1d45c61b2 P 04acd70546f74aabb52e11f0400d733a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:25:48.222082 6350 raft_consensus.cc:482] T 6c4c21ad7c974e3b885565d1d45c61b2 P 04acd70546f74aabb52e11f0400d733a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:25:48.222199 6350 raft_consensus.cc:3037] T 6c4c21ad7c974e3b885565d1d45c61b2 P 04acd70546f74aabb52e11f0400d733a [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:48.223573 6344 ts_tablet_manager.cc:1428] T 6c4c21ad7c974e3b885565d1d45c61b2 P baa87368342f48f3a13217d00339ea1f: Time spent starting tablet: real 0.020s user 0.014s sys 0.005s
I20260811 22:25:48.225342 6350 raft_consensus.cc:504] T 6c4c21ad7c974e3b885565d1d45c61b2 P 04acd70546f74aabb52e11f0400d733a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04acd70546f74aabb52e11f0400d733a" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 37673 } } peers { permanent_uuid: "d71d6d24b717473e80e9cd393d0b8f6c" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 40065 } } peers { permanent_uuid: "baa87368342f48f3a13217d00339ea1f" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 40963 } }
I20260811 22:25:48.225835 6345 ts_tablet_manager.cc:1428] T 6c4c21ad7c974e3b885565d1d45c61b2 P d71d6d24b717473e80e9cd393d0b8f6c: Time spent starting tablet: real 0.023s user 0.010s sys 0.009s
I20260811 22:25:48.226156 6045 heartbeater.cc:503] Master 127.29.106.62:41209 was elected leader, sending a full tablet report...
I20260811 22:25:48.227427 6338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:48.220257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:60995 (local address 127.29.106.1:40963)
0811 22:25:48.220970 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:48.220975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:48.220994 (+ 19us) server_negotiation.cc:407] Connection header received
0811 22:25:48.221062 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:48.221068 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:48.221129 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:48.221207 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:25:48.221652 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:48.222755 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:48.225109 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:48.226271 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:48.226404 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:48.226471 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:48.227068 (+ 597us) server_negotiation.cc:299] Negotiation successful
0811 22:25:48.227192 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563}
I20260811 22:25:48.227963 6350 leader_election.cc:288] T 6c4c21ad7c974e3b885565d1d45c61b2 P 04acd70546f74aabb52e11f0400d733a [CANDIDATE]: Term 1 election: Requested vote from peers d71d6d24b717473e80e9cd393d0b8f6c (127.29.106.2:40065), baa87368342f48f3a13217d00339ea1f (127.29.106.1:40963)
I20260811 22:25:48.228272 6307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:48.220263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:40963 (local address 127.29.106.3:60995)
0811 22:25:48.220505 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:48.220524 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:48.220629 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:48.221244 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:48.221254 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:48.221278 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:25:48.221474 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:48.221487 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:48.222930 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:48.222936 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:48.224903 (+ 1967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:48.224918 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:48.226657 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:48.226663 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:48.226882 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:48.226913 (+ 31us) client_negotiation.cc:769] Sending connection context
0811 22:25:48.227768 (+ 855us) client_negotiation.cc:240] Negotiation successful
0811 22:25:48.227898 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141}
I20260811 22:25:48.228308 5953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c4c21ad7c974e3b885565d1d45c61b2" candidate_uuid: "04acd70546f74aabb52e11f0400d733a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d71d6d24b717473e80e9cd393d0b8f6c"
I20260811 22:25:48.228495 5953 raft_consensus.cc:3037] T 6c4c21ad7c974e3b885565d1d45c61b2 P d71d6d24b717473e80e9cd393d0b8f6c [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:48.229452 5691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c4c21ad7c974e3b885565d1d45c61b2" candidate_uuid: "04acd70546f74aabb52e11f0400d733a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baa87368342f48f3a13217d00339ea1f"
I20260811 22:25:48.229475 5692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c4c21ad7c974e3b885565d1d45c61b2" candidate_uuid: "04acd70546f74aabb52e11f0400d733a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baa87368342f48f3a13217d00339ea1f" is_pre_election: true
I20260811 22:25:48.230070 5692 raft_consensus.cc:2454] T 6c4c21ad7c974e3b885565d1d45c61b2 P baa87368342f48f3a13217d00339ea1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04acd70546f74aabb52e11f0400d733a in term 0.
I20260811 22:25:48.231391 5953 raft_consensus.cc:2454] T 6c4c21ad7c974e3b885565d1d45c61b2 P d71d6d24b717473e80e9cd393d0b8f6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04acd70546f74aabb52e11f0400d733a in term 1.
I20260811 22:25:48.231719 6094 leader_election.cc:302] T 6c4c21ad7c974e3b885565d1d45c61b2 P 04acd70546f74aabb52e11f0400d733a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 04acd70546f74aabb52e11f0400d733a, d71d6d24b717473e80e9cd393d0b8f6c; no voters: baa87368342f48f3a13217d00339ea1f
I20260811 22:25:48.231982 6350 raft_consensus.cc:2781] T 6c4c21ad7c974e3b885565d1d45c61b2 P 04acd70546f74aabb52e11f0400d733a [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:25:48.232146 6350 raft_consensus.cc:686] T 6c4c21ad7c974e3b885565d1d45c61b2 P 04acd70546f74aabb52e11f0400d733a [term 1 LEADER]: Becoming Leader. State: Replica: 04acd70546f74aabb52e11f0400d733a, State: Running, Role: LEADER
I20260811 22:25:48.232568 6350 consensus_queue.cc:237] T 6c4c21ad7c974e3b885565d1d45c61b2 P 04acd70546f74aabb52e11f0400d733a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04acd70546f74aabb52e11f0400d733a" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 37673 } } peers { permanent_uuid: "d71d6d24b717473e80e9cd393d0b8f6c" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 40065 } } peers { permanent_uuid: "baa87368342f48f3a13217d00339ea1f" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 40963 } }
I20260811 22:25:48.236027 5430 catalog_manager.cc:5495] T 6c4c21ad7c974e3b885565d1d45c61b2 P 04acd70546f74aabb52e11f0400d733a reported cstate change: term changed from 0 to 1, leader changed from <none> to 04acd70546f74aabb52e11f0400d733a (127.29.106.3). New cstate: current_term: 1 leader_uuid: "04acd70546f74aabb52e11f0400d733a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04acd70546f74aabb52e11f0400d733a" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 37673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d71d6d24b717473e80e9cd393d0b8f6c" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 40065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "baa87368342f48f3a13217d00339ea1f" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 40963 } health_report { overall_health: UNKNOWN } } }
I20260811 22:25:48.262068 6325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:48.257017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57970 (local address 127.29.106.3:37673)
0811 22:25:48.257188 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:48.257194 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:48.257362 (+ 168us) server_negotiation.cc:407] Connection header received
0811 22:25:48.257570 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:48.257574 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:48.257642 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:48.257758 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:25:48.258133 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:48.258822 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:48.260157 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:48.260549 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:48.260654 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:48.261100 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:25:48.261515 (+ 415us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:25:48.261648 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:48.261751 (+ 103us) server_negotiation.cc:299] Negotiation successful
0811 22:25:48.261828 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260811 22:25:48.269063 5691 raft_consensus.cc:3037] T 6c4c21ad7c974e3b885565d1d45c61b2 P baa87368342f48f3a13217d00339ea1f [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:48.269253 5953 raft_consensus.cc:1261] T 6c4c21ad7c974e3b885565d1d45c61b2 P d71d6d24b717473e80e9cd393d0b8f6c [term 1 FOLLOWER]: Refusing update from remote peer 04acd70546f74aabb52e11f0400d733a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:25:48.269999 6352 consensus_queue.cc:1035] T 6c4c21ad7c974e3b885565d1d45c61b2 P 04acd70546f74aabb52e11f0400d733a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d71d6d24b717473e80e9cd393d0b8f6c" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 40065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:25:48.271449 5691 raft_consensus.cc:1261] T 6c4c21ad7c974e3b885565d1d45c61b2 P baa87368342f48f3a13217d00339ea1f [term 1 FOLLOWER]: Refusing update from remote peer 04acd70546f74aabb52e11f0400d733a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:25:48.272217 6352 consensus_queue.cc:1035] T 6c4c21ad7c974e3b885565d1d45c61b2 P 04acd70546f74aabb52e11f0400d733a [LEADER]: Connected to new peer: Peer: permanent_uuid: "baa87368342f48f3a13217d00339ea1f" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 40963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:25:48.278668 6358 mvcc.cc:205] Tried to move back new op lower bound from 7317451359302270976 to 7317451359163842560. Current Snapshot: MvccSnapshot[applied={T|T < 7317451359302270976}]
I20260811 22:25:48.282424 6359 mvcc.cc:205] Tried to move back new op lower bound from 7317451359302270976 to 7317451359163842560. Current Snapshot: MvccSnapshot[applied={T|T < 7317451359302270976 or (T in {7317451359302270976})}]
I20260811 22:25:48.301394 5430 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:25:48.301610 5430 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:25:48.305250 5430 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a6a63a5163044443b0301e1ac8172a20: Sending DeleteTablet for 3 replicas of tablet 6c4c21ad7c974e3b885565d1d45c61b2
I20260811 22:25:48.306268 6176 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c4c21ad7c974e3b885565d1d45c61b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:25:48 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:57966
I20260811 22:25:48.306407 5912 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c4c21ad7c974e3b885565d1d45c61b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:25:48 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:54698
I20260811 22:25:48.306602 5655 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c4c21ad7c974e3b885565d1d45c61b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:25:48 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:55324
I20260811 22:25:48.306656 5653 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c4c21ad7c974e3b885565d1d45c61b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:25:48 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:55324
I20260811 22:25:48.308924 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 5539
I20260811 22:25:48.316351 6384 tablet_replica.cc:330] T 6c4c21ad7c974e3b885565d1d45c61b2 P baa87368342f48f3a13217d00339ea1f: stopping tablet replica
I20260811 22:25:48.316829 6384 raft_consensus.cc:2229] T 6c4c21ad7c974e3b885565d1d45c61b2 P baa87368342f48f3a13217d00339ea1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:25:48.317286 6384 raft_consensus.cc:2258] T 6c4c21ad7c974e3b885565d1d45c61b2 P baa87368342f48f3a13217d00339ea1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:25:48.321345 6382 tablet_replica.cc:330] T 6c4c21ad7c974e3b885565d1d45c61b2 P d71d6d24b717473e80e9cd393d0b8f6c: stopping tablet replica
I20260811 22:25:48.321827 6382 raft_consensus.cc:2229] T 6c4c21ad7c974e3b885565d1d45c61b2 P d71d6d24b717473e80e9cd393d0b8f6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:25:48.322397 6382 raft_consensus.cc:2258] T 6c4c21ad7c974e3b885565d1d45c61b2 P d71d6d24b717473e80e9cd393d0b8f6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:25:48.323184 6381 tablet_replica.cc:330] T 6c4c21ad7c974e3b885565d1d45c61b2 P 04acd70546f74aabb52e11f0400d733a: stopping tablet replica
I20260811 22:25:48.323650 6381 raft_consensus.cc:2229] T 6c4c21ad7c974e3b885565d1d45c61b2 P 04acd70546f74aabb52e11f0400d733a [term 1 LEADER]: Raft consensus shutting down.
I20260811 22:25:48.324261 6381 raft_consensus.cc:2258] T 6c4c21ad7c974e3b885565d1d45c61b2 P 04acd70546f74aabb52e11f0400d733a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:25:48.324916 6384 ts_tablet_manager.cc:1905] T 6c4c21ad7c974e3b885565d1d45c61b2 P baa87368342f48f3a13217d00339ea1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:25:48.326117 6382 ts_tablet_manager.cc:1905] T 6c4c21ad7c974e3b885565d1d45c61b2 P d71d6d24b717473e80e9cd393d0b8f6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:25:48.326452 6381 ts_tablet_manager.cc:1905] T 6c4c21ad7c974e3b885565d1d45c61b2 P 04acd70546f74aabb52e11f0400d733a: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 11 22:25:48 dist-test-slave-gjrz krb5kdc[5347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.1@KRBTEST.COM
I20260811 22:25:48.332065 6384 ts_tablet_manager.cc:1918] T 6c4c21ad7c974e3b885565d1d45c61b2 P baa87368342f48f3a13217d00339ea1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:25:48.332240 6384 log.cc:1196] T 6c4c21ad7c974e3b885565d1d45c61b2 P baa87368342f48f3a13217d00339ea1f: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-0/wal/wals/6c4c21ad7c974e3b885565d1d45c61b2
I20260811 22:25:48.332810 6384 ts_tablet_manager.cc:1939] T 6c4c21ad7c974e3b885565d1d45c61b2 P baa87368342f48f3a13217d00339ea1f: Deleting consensus metadata
I20260811 22:25:48.333642 6381 ts_tablet_manager.cc:1918] T 6c4c21ad7c974e3b885565d1d45c61b2 P 04acd70546f74aabb52e11f0400d733a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:25:48.333885 6381 log.cc:1196] T 6c4c21ad7c974e3b885565d1d45c61b2 P 04acd70546f74aabb52e11f0400d733a: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-2/wal/wals/6c4c21ad7c974e3b885565d1d45c61b2
I20260811 22:25:48.334548 6381 ts_tablet_manager.cc:1939] T 6c4c21ad7c974e3b885565d1d45c61b2 P 04acd70546f74aabb52e11f0400d733a: Deleting consensus metadata
I20260811 22:25:48.334790 5400 catalog_manager.cc:4841] TS baa87368342f48f3a13217d00339ea1f (127.29.106.1:40963): tablet 6c4c21ad7c974e3b885565d1d45c61b2 (table test-table [id=88f7f6ba704943cb95ad7534614df41a]) successfully deleted
W20260811 22:25:48.336014 5400 catalog_manager.cc:4808] TS baa87368342f48f3a13217d00339ea1f (127.29.106.1:40963): delete failed for tablet 6c4c21ad7c974e3b885565d1d45c61b2 because the tablet was not found. No further retry: Not found: Tablet not found: 6c4c21ad7c974e3b885565d1d45c61b2
I20260811 22:25:48.336392 5404 catalog_manager.cc:4841] TS 04acd70546f74aabb52e11f0400d733a (127.29.106.3:37673): tablet 6c4c21ad7c974e3b885565d1d45c61b2 (table test-table [id=88f7f6ba704943cb95ad7534614df41a]) successfully deleted
I20260811 22:25:48.337076 6338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:48.310693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55340 (local address 127.29.106.1:40963)
0811 22:25:48.310875 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:48.310882 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:48.315752 (+ 4870us) server_negotiation.cc:407] Connection header received
0811 22:25:48.315932 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:48.315938 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:48.316031 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:48.316155 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:48.320926 (+ 4771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:48.322181 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:48.323907 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:48.324348 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:48.324464 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:48.329568 (+ 5104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:48.329583 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:48.329601 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:48.329641 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:48.333542 (+ 3901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:48.334425 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:48.334434 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:48.334438 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:48.334626 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:48.335084 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:48.335088 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:48.335091 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:48.335712 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:48.336012 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:48.336608 (+ 596us) server_negotiation.cc:299] Negotiation successful
0811 22:25:48.336827 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260811 22:25:48.338340 5761 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:25:48.341480 6382 ts_tablet_manager.cc:1918] T 6c4c21ad7c974e3b885565d1d45c61b2 P d71d6d24b717473e80e9cd393d0b8f6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:25:48.341683 6382 log.cc:1196] T 6c4c21ad7c974e3b885565d1d45c61b2 P d71d6d24b717473e80e9cd393d0b8f6c: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786487105757681-30120-0/minicluster-data/ts-1/wal/wals/6c4c21ad7c974e3b885565d1d45c61b2
I20260811 22:25:48.342237 6382 ts_tablet_manager.cc:1939] T 6c4c21ad7c974e3b885565d1d45c61b2 P d71d6d24b717473e80e9cd393d0b8f6c: Deleting consensus metadata
I20260811 22:25:48.344008 5404 catalog_manager.cc:4841] TS d71d6d24b717473e80e9cd393d0b8f6c (127.29.106.2:40065): tablet 6c4c21ad7c974e3b885565d1d45c61b2 (table test-table [id=88f7f6ba704943cb95ad7534614df41a]) successfully deleted
I20260811 22:25:48.427062 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 5539
I20260811 22:25:48.456278 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 5783
Aug 11 22:25:48 dist-test-slave-gjrz krb5kdc[5347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.2@KRBTEST.COM
I20260811 22:25:48.472150 6340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:48.457141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54700 (local address 127.29.106.2:40065)
0811 22:25:48.457360 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:48.457366 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:48.457388 (+ 22us) server_negotiation.cc:407] Connection header received
0811 22:25:48.457459 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:48.457464 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:48.457544 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:48.457654 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:48.458786 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:48.459967 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:48.461207 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:48.461828 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:48.461978 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:48.465878 (+ 3900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:48.465906 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:48.465928 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:48.465979 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:48.469511 (+ 3532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:48.470095 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:48.470104 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:48.470109 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:48.470263 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:48.470564 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:48.470569 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:48.470572 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:48.471048 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:48.471359 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:48.471549 (+ 190us) server_negotiation.cc:299] Negotiation successful
0811 22:25:48.471789 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260811 22:25:48.472956 6029 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:25:48.560150 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 5783
I20260811 22:25:48.589358 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 6050
Aug 11 22:25:48 dist-test-slave-gjrz krb5kdc[5347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.3@KRBTEST.COM
I20260811 22:25:48.606544 6325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:48.590190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57980 (local address 127.29.106.3:37673)
0811 22:25:48.590351 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:48.590356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:48.590429 (+ 73us) server_negotiation.cc:407] Connection header received
0811 22:25:48.590632 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:48.590636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:48.590695 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:48.590779 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:48.592512 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:48.593309 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:48.595122 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:48.595611 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:48.595703 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:48.600864 (+ 5161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:48.600878 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:48.600894 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:48.600928 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:48.603810 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:48.604574 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:48.604581 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:48.604583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:48.604729 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:48.605129 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:48.605132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:48.605134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:48.605469 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:48.605705 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:48.606024 (+ 319us) server_negotiation.cc:299] Negotiation successful
0811 22:25:48.606205 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260811 22:25:48.607250 6292 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:25:48.725854 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 6050
I20260811 22:25:48.751997 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 5375
I20260811 22:25:48.753105 5489 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:25:48.859992 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 5375
2026-08-11T22:25:48Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4158 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-11T22:25:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:25:48Z Disabled control of system clock
I20260811 22:25:48.934233 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:46681
--webserver_interface=127.29.106.62
--webserver_port=0
--builtin_ntp_servers=127.29.106.20:38821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:46681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260811 22:25:49.251907 6404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:49.252173 6404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:49.252262 6404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:49.259219 6404 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:25:49.259335 6404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:49.259387 6404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:49.259441 6404 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:25:49.259485 6404 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:25:49.268181 6404 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:38821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:46681
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:46681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.106.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:25:49.269323 6404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:49.271006 6404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:49.282289 6420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:49.282292 6419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:49.282482 6422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:49.282859 6404 server_base.cc:1034] running on GCE node
I20260811 22:25:49.283679 6404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:49.284901 6404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:49.286099 6404 hybrid_clock.cc:648] HybridClock initialized: now 1786487149286063 us; error 26 us; skew 500 ppm
I20260811 22:25:49.286422 6404 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 22:25:49.295918 6404 webserver.cc:460] Webserver started at http://127.29.106.62:44529/ using document root <none> and password file <none>
I20260811 22:25:49.296571 6404 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:49.296687 6404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:49.297006 6404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:49.299876 6404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "2af2db0d30de47208149ffda390a641a"
format_stamp: "Formatted at 2026-08-11 22:25:49 on dist-test-slave-gjrz"
server_key: "1694650eeb577137931d1f4639ee5923"
server_key_iv: "befeecbb7ed2fbf5235d1159e4cdf5b0"
server_key_version: "encryptionkey@0"
I20260811 22:25:49.300647 6404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "2af2db0d30de47208149ffda390a641a"
format_stamp: "Formatted at 2026-08-11 22:25:49 on dist-test-slave-gjrz"
server_key: "1694650eeb577137931d1f4639ee5923"
server_key_iv: "befeecbb7ed2fbf5235d1159e4cdf5b0"
server_key_version: "encryptionkey@0"
I20260811 22:25:49.305519 6404 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260811 22:25:49.308744 6431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:49.309940 6404 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 22:25:49.310088 6404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "2af2db0d30de47208149ffda390a641a"
format_stamp: "Formatted at 2026-08-11 22:25:49 on dist-test-slave-gjrz"
server_key: "1694650eeb577137931d1f4639ee5923"
server_key_iv: "befeecbb7ed2fbf5235d1159e4cdf5b0"
server_key_version: "encryptionkey@0"
I20260811 22:25:49.310278 6404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:49.315716 6404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:49.316404 6404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:49.316681 6404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:49.334182 6404 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.62:46681
I20260811 22:25:49.334218 6526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.62:46681 every 8 connection(s)
I20260811 22:25:49.335392 6404 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:25:49.338817 6531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:49.340494 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 6404
I20260811 22:25:49.340716 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:25:49.341512 30120 external_mini_cluster.cc:1428] Setting key 3cbe4f24c17d5b1db937356c13c47309
I20260811 22:25:49.349505 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:49.343505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43434 (local address 127.29.106.62:46681)
0811 22:25:49.344639 (+ 1134us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:49.344649 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:49.344698 (+ 49us) server_negotiation.cc:407] Connection header received
0811 22:25:49.346330 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:49.346350 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:49.346708 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:49.347001 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:49.348060 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:49.348066 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:49.348078 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:25:49.348094 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:49.348154 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:25:49.348170 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:49.348332 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:49.348421 (+ 89us) server_negotiation.cc:299] Negotiation successful
0811 22:25:49.348653 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":421,"threads_started":1}
I20260811 22:25:49.350757 6531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2af2db0d30de47208149ffda390a641a: Bootstrap starting.
I20260811 22:25:49.353632 6531 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2af2db0d30de47208149ffda390a641a: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:49.354601 6531 log.cc:824] T 00000000000000000000000000000000 P 2af2db0d30de47208149ffda390a641a: Log is configured to *not* fsync() on all Append() calls
I20260811 22:25:49.356995 6531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2af2db0d30de47208149ffda390a641a: No bootstrap required, opened a new log
I20260811 22:25:49.363710 6531 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2af2db0d30de47208149ffda390a641a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2af2db0d30de47208149ffda390a641a" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46681 } }
I20260811 22:25:49.364012 6531 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2af2db0d30de47208149ffda390a641a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:25:49.364120 6531 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2af2db0d30de47208149ffda390a641a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2af2db0d30de47208149ffda390a641a, State: Initialized, Role: FOLLOWER
I20260811 22:25:49.364640 6531 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2af2db0d30de47208149ffda390a641a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2af2db0d30de47208149ffda390a641a" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46681 } }
I20260811 22:25:49.364846 6531 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2af2db0d30de47208149ffda390a641a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:25:49.364974 6531 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2af2db0d30de47208149ffda390a641a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:25:49.365113 6531 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2af2db0d30de47208149ffda390a641a [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:49.367125 6531 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2af2db0d30de47208149ffda390a641a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2af2db0d30de47208149ffda390a641a" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46681 } }
I20260811 22:25:49.367631 6531 leader_election.cc:302] T 00000000000000000000000000000000 P 2af2db0d30de47208149ffda390a641a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2af2db0d30de47208149ffda390a641a; no voters:
I20260811 22:25:49.368167 6531 leader_election.cc:288] T 00000000000000000000000000000000 P 2af2db0d30de47208149ffda390a641a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:25:49.368289 6540 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2af2db0d30de47208149ffda390a641a [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:25:49.368955 6540 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2af2db0d30de47208149ffda390a641a [term 1 LEADER]: Becoming Leader. State: Replica: 2af2db0d30de47208149ffda390a641a, State: Running, Role: LEADER
I20260811 22:25:49.369467 6540 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2af2db0d30de47208149ffda390a641a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2af2db0d30de47208149ffda390a641a" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46681 } }
I20260811 22:25:49.369975 6531 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2af2db0d30de47208149ffda390a641a [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:25:49.372007 6541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2af2db0d30de47208149ffda390a641a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2af2db0d30de47208149ffda390a641a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2af2db0d30de47208149ffda390a641a" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46681 } } }
I20260811 22:25:49.372561 6541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2af2db0d30de47208149ffda390a641a [sys.catalog]: This master's current role is: LEADER
I20260811 22:25:49.372278 6545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2af2db0d30de47208149ffda390a641a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2af2db0d30de47208149ffda390a641a. Latest consensus state: current_term: 1 leader_uuid: "2af2db0d30de47208149ffda390a641a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2af2db0d30de47208149ffda390a641a" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46681 } } }
I20260811 22:25:49.372905 6545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2af2db0d30de47208149ffda390a641a [sys.catalog]: This master's current role is: LEADER
I20260811 22:25:49.374802 6556 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:25:49.379787 6556 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:25:49.388113 6556 catalog_manager.cc:1347] Generated new cluster ID: 9d23631f8f7b411f90a8a9cc6153cff5
I20260811 22:25:49.388212 6556 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:25:49.404690 6556 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:25:49.405145 6556 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:25:49.425599 6556 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2af2db0d30de47208149ffda390a641a: Generated new TSK 0
I20260811 22:25:49.426329 6556 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:25:49.445444 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.1:0
--local_ip_for_outbound_sockets=127.29.106.1
--webserver_interface=127.29.106.1
--webserver_port=0
--tserver_master_addrs=127.29.106.62:46681
--builtin_ntp_servers=127.29.106.20:38821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260811 22:25:49.765230 6578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:49.765491 6578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:49.765617 6578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:49.773110 6578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:49.773203 6578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:49.773344 6578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.1
I20260811 22:25:49.782189 6578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:38821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.106.1
--webserver_port=0
--tserver_master_addrs=127.29.106.62:46681
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.1
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:25:49.783398 6578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:49.785106 6578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:49.797413 6591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:49.797502 6595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:49.797533 6592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:49.798209 6578 server_base.cc:1034] running on GCE node
I20260811 22:25:49.798748 6578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:49.799654 6578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:49.800859 6578 hybrid_clock.cc:648] HybridClock initialized: now 1786487149800801 us; error 46 us; skew 500 ppm
I20260811 22:25:49.801158 6578 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 22:25:49.803304 6578 webserver.cc:460] Webserver started at http://127.29.106.1:46875/ using document root <none> and password file <none>
I20260811 22:25:49.803990 6578 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:49.804095 6578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:49.804406 6578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:49.807093 6578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-0/data/instance:
uuid: "f83b5295dfc64676b328137aa7d59990"
format_stamp: "Formatted at 2026-08-11 22:25:49 on dist-test-slave-gjrz"
server_key: "19a7c8ad0bfe8ddf1ee09a2b52e08e7e"
server_key_iv: "c006df95c3e6ae395c878f204bd98dde"
server_key_version: "encryptionkey@0"
I20260811 22:25:49.807917 6578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance:
uuid: "f83b5295dfc64676b328137aa7d59990"
format_stamp: "Formatted at 2026-08-11 22:25:49 on dist-test-slave-gjrz"
server_key: "19a7c8ad0bfe8ddf1ee09a2b52e08e7e"
server_key_iv: "c006df95c3e6ae395c878f204bd98dde"
server_key_version: "encryptionkey@0"
I20260811 22:25:49.812803 6578 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260811 22:25:49.816243 6605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:49.817790 6578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260811 22:25:49.817925 6578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-0/wal
uuid: "f83b5295dfc64676b328137aa7d59990"
format_stamp: "Formatted at 2026-08-11 22:25:49 on dist-test-slave-gjrz"
server_key: "19a7c8ad0bfe8ddf1ee09a2b52e08e7e"
server_key_iv: "c006df95c3e6ae395c878f204bd98dde"
server_key_version: "encryptionkey@0"
I20260811 22:25:49.818111 6578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:49.823480 6578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:49.824186 6578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:49.824410 6578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:49.825222 6578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:25:49.826519 6578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:25:49.826606 6578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:49.826684 6578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:25:49.826737 6578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:49.856535 6578 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.1:43483
I20260811 22:25:49.856602 6830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.1:43483 every 8 connection(s)
I20260811 22:25:49.857715 6578 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
I20260811 22:25:49.865393 6833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:49.860036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:37649 (local address 127.29.106.62:46681)
0811 22:25:49.860444 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:49.860449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:49.861522 (+ 1073us) server_negotiation.cc:407] Connection header received
0811 22:25:49.863216 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:49.863222 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:49.863313 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:49.863414 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:49.864652 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:49.864656 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:49.864660 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:25:49.864677 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:49.864720 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:25:49.864732 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:49.864839 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:49.865028 (+ 189us) server_negotiation.cc:299] Negotiation successful
0811 22:25:49.865104 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":204,"threads_started":1}
I20260811 22:25:49.866283 6834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:49.860478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:46681 (local address 127.29.106.1:37649)
0811 22:25:49.861359 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:49.861402 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:49.862934 (+ 1532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:49.863659 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:49.863668 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:49.863758 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:49.864336 (+ 578us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 22:25:49.864339 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:49.864377 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 22:25:49.864464 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:49.864824 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:49.864835 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:49.864844 (+ 9us) client_negotiation.cc:769] Sending connection context
0811 22:25:49.865116 (+ 272us) client_negotiation.cc:240] Negotiation successful
0811 22:25:49.865365 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":248,"threads_started":1}
I20260811 22:25:49.866483 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 6578
I20260811 22:25:49.866693 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance
I20260811 22:25:49.867449 30120 external_mini_cluster.cc:1428] Setting key 338de28721d4a7f534cab00178caa454
I20260811 22:25:49.867825 6831 heartbeater.cc:347] Connected to a master server at 127.29.106.62:46681
I20260811 22:25:49.868104 6831 heartbeater.cc:464] Registering TS with master...
I20260811 22:25:49.868665 6831 heartbeater.cc:511] Master 127.29.106.62:46681 requested a full tablet report, sending...
I20260811 22:25:49.870172 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.2:0
--local_ip_for_outbound_sockets=127.29.106.2
--webserver_interface=127.29.106.2
--webserver_port=0
--tserver_master_addrs=127.29.106.62:46681
--builtin_ntp_servers=127.29.106.20:38821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260811 22:25:49.870726 6457 ts_manager.cc:194] Registered new tserver with Master: f83b5295dfc64676b328137aa7d59990 (127.29.106.1:43483)
W20260811 22:25:50.185509 6835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:50.185755 6835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:50.185844 6835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:50.192802 6835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:50.192890 6835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:50.192994 6835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.2
I20260811 22:25:50.201550 6835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:38821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.106.2
--webserver_port=0
--tserver_master_addrs=127.29.106.62:46681
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.2
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:25:50.202656 6835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:50.204349 6835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:50.216886 6844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:50.216917 6847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:50.216846 6845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:50.217729 6835 server_base.cc:1034] running on GCE node
I20260811 22:25:50.218250 6835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:50.219132 6835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:50.220325 6835 hybrid_clock.cc:648] HybridClock initialized: now 1786487150220274 us; error 46 us; skew 500 ppm
I20260811 22:25:50.220611 6835 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 22:25:50.222738 6835 webserver.cc:460] Webserver started at http://127.29.106.2:36435/ using document root <none> and password file <none>
I20260811 22:25:50.223402 6835 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:50.223505 6835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:50.223812 6835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:50.226444 6835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-1/data/instance:
uuid: "b81c4e7f2fd64049a293624f0e0911f6"
format_stamp: "Formatted at 2026-08-11 22:25:50 on dist-test-slave-gjrz"
server_key: "44ea7dbf138e29f99ec3c4a667f7f379"
server_key_iv: "0d776c0a056c5ba94f3ca8dd58f72ed2"
server_key_version: "encryptionkey@0"
I20260811 22:25:50.227190 6835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance:
uuid: "b81c4e7f2fd64049a293624f0e0911f6"
format_stamp: "Formatted at 2026-08-11 22:25:50 on dist-test-slave-gjrz"
server_key: "44ea7dbf138e29f99ec3c4a667f7f379"
server_key_iv: "0d776c0a056c5ba94f3ca8dd58f72ed2"
server_key_version: "encryptionkey@0"
I20260811 22:25:50.232029 6835 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260811 22:25:50.235229 6857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:50.236531 6835 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260811 22:25:50.236668 6835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-1/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-1/wal
uuid: "b81c4e7f2fd64049a293624f0e0911f6"
format_stamp: "Formatted at 2026-08-11 22:25:50 on dist-test-slave-gjrz"
server_key: "44ea7dbf138e29f99ec3c4a667f7f379"
server_key_iv: "0d776c0a056c5ba94f3ca8dd58f72ed2"
server_key_version: "encryptionkey@0"
I20260811 22:25:50.236861 6835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:50.242237 6835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:50.242956 6835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:50.243206 6835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:50.244052 6835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:25:50.245376 6835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:25:50.245476 6835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:50.245580 6835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:25:50.245651 6835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:50.275486 6835 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.2:32951
I20260811 22:25:50.275576 7076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.2:32951 every 8 connection(s)
I20260811 22:25:50.276680 6835 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
I20260811 22:25:50.280300 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 6835
I20260811 22:25:50.280491 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance
I20260811 22:25:50.281035 30120 external_mini_cluster.cc:1428] Setting key 6ec0579539a403d3b4e9ee8c4dddd953
I20260811 22:25:50.283941 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.3:0
--local_ip_for_outbound_sockets=127.29.106.3
--webserver_interface=127.29.106.3
--webserver_port=0
--tserver_master_addrs=127.29.106.62:46681
--builtin_ntp_servers=127.29.106.20:38821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260811 22:25:50.285071 6833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:50.279014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:33475 (local address 127.29.106.62:46681)
0811 22:25:50.279174 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:50.279179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:50.280773 (+ 1594us) server_negotiation.cc:407] Connection header received
0811 22:25:50.282930 (+ 2157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:50.282936 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:50.282992 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:50.283082 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:50.284342 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:50.284349 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:50.284353 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:25:50.284372 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:50.284413 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:25:50.284428 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:50.284541 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:50.284716 (+ 175us) server_negotiation.cc:299] Negotiation successful
0811 22:25:50.284778 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260811 22:25:50.285816 7081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:50.279487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:46681 (local address 127.29.106.2:33475)
0811 22:25:50.280602 (+ 1115us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:50.280650 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:50.282615 (+ 1965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:50.283324 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:50.283332 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:50.283420 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:50.284026 (+ 606us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 22:25:50.284029 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:50.284071 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 22:25:50.284176 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:50.284543 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:50.284551 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:50.284556 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 22:25:50.284761 (+ 205us) client_negotiation.cc:240] Negotiation successful
0811 22:25:50.284940 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":286,"threads_started":1}
I20260811 22:25:50.287053 7079 heartbeater.cc:347] Connected to a master server at 127.29.106.62:46681
I20260811 22:25:50.287353 7079 heartbeater.cc:464] Registering TS with master...
I20260811 22:25:50.287892 7079 heartbeater.cc:511] Master 127.29.106.62:46681 requested a full tablet report, sending...
I20260811 22:25:50.289213 6457 ts_manager.cc:194] Registered new tserver with Master: b81c4e7f2fd64049a293624f0e0911f6 (127.29.106.2:32951)
W20260811 22:25:50.607930 7085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:50.608161 7085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:50.608253 7085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:50.615120 7085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:50.615207 7085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:50.615335 7085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.3
I20260811 22:25:50.623970 7085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:38821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.106.3
--webserver_port=0
--tserver_master_addrs=127.29.106.62:46681
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.3
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:25:50.625046 7085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:50.626654 7085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:50.638846 7097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:50.638846 7101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:50.638888 7104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:50.639679 7085 server_base.cc:1034] running on GCE node
I20260811 22:25:50.640245 7085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:50.641165 7085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:50.642359 7085 hybrid_clock.cc:648] HybridClock initialized: now 1786487150642316 us; error 28 us; skew 500 ppm
I20260811 22:25:50.642655 7085 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 22:25:50.644726 7085 webserver.cc:460] Webserver started at http://127.29.106.3:44415/ using document root <none> and password file <none>
I20260811 22:25:50.645344 7085 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:50.645440 7085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:50.645756 7085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:50.648481 7085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-2/data/instance:
uuid: "46d01e363bda4bef9d9949821d670926"
format_stamp: "Formatted at 2026-08-11 22:25:50 on dist-test-slave-gjrz"
server_key: "37ba603febbe3a1539847677ebfd6b32"
server_key_iv: "eab393286e305d8821f2860bf6fdce20"
server_key_version: "encryptionkey@0"
I20260811 22:25:50.649214 7085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance:
uuid: "46d01e363bda4bef9d9949821d670926"
format_stamp: "Formatted at 2026-08-11 22:25:50 on dist-test-slave-gjrz"
server_key: "37ba603febbe3a1539847677ebfd6b32"
server_key_iv: "eab393286e305d8821f2860bf6fdce20"
server_key_version: "encryptionkey@0"
I20260811 22:25:50.653867 7085 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 22:25:50.657002 7113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:50.658154 7085 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260811 22:25:50.658295 7085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-2/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-2/wal
uuid: "46d01e363bda4bef9d9949821d670926"
format_stamp: "Formatted at 2026-08-11 22:25:50 on dist-test-slave-gjrz"
server_key: "37ba603febbe3a1539847677ebfd6b32"
server_key_iv: "eab393286e305d8821f2860bf6fdce20"
server_key_version: "encryptionkey@0"
I20260811 22:25:50.658483 7085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:50.663705 7085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:50.664414 7085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:50.664645 7085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:50.665450 7085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:25:50.666713 7085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:25:50.666806 7085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:50.666913 7085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:25:50.666981 7085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:50.694922 7085 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.3:38165
I20260811 22:25:50.694962 7321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.3:38165 every 8 connection(s)
I20260811 22:25:50.696141 7085 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
I20260811 22:25:50.703471 6833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:50.698503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:44795 (local address 127.29.106.62:46681)
0811 22:25:50.698672 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:50.698676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:50.700008 (+ 1332us) server_negotiation.cc:407] Connection header received
0811 22:25:50.701613 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:50.701617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:50.701671 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:50.701760 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:50.702876 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:50.702880 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:50.702884 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:25:50.702898 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:50.702937 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:25:50.702948 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:50.703046 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:50.703173 (+ 127us) server_negotiation.cc:299] Negotiation successful
0811 22:25:50.703221 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260811 22:25:50.703907 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 7085
I20260811 22:25:50.704101 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance
I20260811 22:25:50.704583 7328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:50.698986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:46681 (local address 127.29.106.3:44795)
0811 22:25:50.699861 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:50.699906 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:50.701372 (+ 1466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:50.701948 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:50.701955 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:50.702038 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:50.702611 (+ 573us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 22:25:50.702613 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:50.702648 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 22:25:50.702727 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:50.703019 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:50.703025 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:50.703030 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 22:25:50.703234 (+ 204us) client_negotiation.cc:240] Negotiation successful
0811 22:25:50.703459 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":287,"threads_started":1}
I20260811 22:25:50.704640 30120 external_mini_cluster.cc:1428] Setting key 1d904a15c194103f13ae5c5dc1d74118
I20260811 22:25:50.705896 7322 heartbeater.cc:347] Connected to a master server at 127.29.106.62:46681
I20260811 22:25:50.706177 7322 heartbeater.cc:464] Registering TS with master...
I20260811 22:25:50.706743 7322 heartbeater.cc:511] Master 127.29.106.62:46681 requested a full tablet report, sending...
I20260811 22:25:50.708076 6457 ts_manager.cc:194] Registered new tserver with Master: 46d01e363bda4bef9d9949821d670926 (127.29.106.3:38165)
I20260811 22:25:50.719106 30120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 22:25:50.725456 6833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:50.723393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34468 (local address 127.29.106.62:46681)
0811 22:25:50.723545 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:50.723550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:50.724075 (+ 525us) server_negotiation.cc:407] Connection header received
0811 22:25:50.724183 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:50.724186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:50.724236 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:50.724311 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:50.725032 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:50.725036 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:50.725039 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:25:50.725053 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:50.725083 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:25:50.725094 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:50.725182 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:50.725212 (+ 30us) server_negotiation.cc:299] Negotiation successful
0811 22:25:50.725254 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260811 22:25:50.728896 6457 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34468:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:25:50.735632 6457 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:25:50.752588 7351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:50.747190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47214 (local address 127.29.106.1:43483)
0811 22:25:50.749999 (+ 2809us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:50.750007 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:50.750036 (+ 29us) server_negotiation.cc:407] Connection header received
0811 22:25:50.750118 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:50.750125 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:50.750345 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:50.750488 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:50.751787 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:50.751793 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:50.751803 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:25:50.751825 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:50.751869 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:25:50.751884 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:50.752010 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:50.752186 (+ 176us) server_negotiation.cc:299] Negotiation successful
0811 22:25:50.752279 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2663,"thread_start_us":2549,"threads_started":1}
I20260811 22:25:50.752688 7355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:50.746609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:43483 (local address 127.0.0.1:47214)
0811 22:25:50.749435 (+ 2826us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:50.749455 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:50.749556 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:50.750543 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:50.750562 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:50.750651 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:50.751521 (+ 870us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 22:25:50.751524 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:50.751566 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 22:25:50.751649 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:50.751993 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:50.752004 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:50.752012 (+ 8us) client_negotiation.cc:769] Sending connection context
0811 22:25:50.752178 (+ 166us) client_negotiation.cc:240] Negotiation successful
0811 22:25:50.752264 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2681,"thread_start_us":1838,"threads_started":1}
I20260811 22:25:50.753245 7348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:50.746614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49176 (local address 127.29.106.2:32951)
0811 22:25:50.748288 (+ 1674us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:50.748298 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:50.749165 (+ 867us) server_negotiation.cc:407] Connection header received
0811 22:25:50.749240 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:50.749248 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:50.750481 (+ 1233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:50.750632 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:50.752067 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:50.752076 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:50.752089 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:25:50.752117 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:50.752175 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:25:50.752196 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:50.752342 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:50.752801 (+ 459us) server_negotiation.cc:299] Negotiation successful
0811 22:25:50.752898 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1494,"thread_start_us":1991,"threads_started":1}
I20260811 22:25:50.753697 7349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:50.746449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.2:32951 (local address 127.0.0.1:49176)
0811 22:25:50.748850 (+ 2401us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:50.748890 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:50.749035 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:50.751054 (+ 2019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:50.751059 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:50.751080 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:50.751853 (+ 773us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 22:25:50.751857 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:50.751871 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 22:25:50.751912 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:50.752407 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:50.752412 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:50.752416 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 22:25:50.752578 (+ 162us) client_negotiation.cc:240] Negotiation successful
0811 22:25:50.752635 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2171,"thread_start_us":3592,"threads_started":1}
I20260811 22:25:50.755541 7356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:50.748758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40754 (local address 127.29.106.3:38165)
0811 22:25:50.751853 (+ 3095us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:50.751861 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:50.751890 (+ 29us) server_negotiation.cc:407] Connection header received
0811 22:25:50.751955 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:50.751962 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:50.752159 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:50.752294 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:50.754693 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:50.754701 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:50.754715 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:25:50.754734 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:50.754798 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:25:50.754820 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:50.754988 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:50.755050 (+ 62us) server_negotiation.cc:299] Negotiation successful
0811 22:25:50.755141 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2931,"thread_start_us":2806,"threads_started":1}
I20260811 22:25:50.755564 7347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:50.748187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.3:38165 (local address 127.0.0.1:40754)
0811 22:25:50.750291 (+ 2104us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:50.750308 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:50.750394 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:50.753970 (+ 3576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:50.753977 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:50.753996 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:50.754513 (+ 517us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 22:25:50.754516 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:50.754528 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 22:25:50.754561 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:50.754937 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:50.754942 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:50.754947 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 22:25:50.755076 (+ 129us) client_negotiation.cc:240] Negotiation successful
0811 22:25:50.755126 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1998,"thread_start_us":965,"threads_started":1}
I20260811 22:25:50.756948 6697 tablet_service.cc:1467] Processing CreateTablet for tablet c84d30630cfd4d53ac6e3e0cd9bfe0df (DEFAULT_TABLE table=test-table [id=3640a4e64a5e4522928c32067e8d7509]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:25:50.757290 6948 tablet_service.cc:1467] Processing CreateTablet for tablet c84d30630cfd4d53ac6e3e0cd9bfe0df (DEFAULT_TABLE table=test-table [id=3640a4e64a5e4522928c32067e8d7509]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:25:50.758490 6948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c84d30630cfd4d53ac6e3e0cd9bfe0df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:50.758499 6697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c84d30630cfd4d53ac6e3e0cd9bfe0df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:50.759042 7210 tablet_service.cc:1467] Processing CreateTablet for tablet c84d30630cfd4d53ac6e3e0cd9bfe0df (DEFAULT_TABLE table=test-table [id=3640a4e64a5e4522928c32067e8d7509]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:25:50.760339 7210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c84d30630cfd4d53ac6e3e0cd9bfe0df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:50.769588 7358 tablet_bootstrap.cc:492] T c84d30630cfd4d53ac6e3e0cd9bfe0df P b81c4e7f2fd64049a293624f0e0911f6: Bootstrap starting.
I20260811 22:25:50.772727 7359 tablet_bootstrap.cc:492] T c84d30630cfd4d53ac6e3e0cd9bfe0df P f83b5295dfc64676b328137aa7d59990: Bootstrap starting.
I20260811 22:25:50.773017 7358 tablet_bootstrap.cc:654] T c84d30630cfd4d53ac6e3e0cd9bfe0df P b81c4e7f2fd64049a293624f0e0911f6: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:50.774292 7358 log.cc:824] T c84d30630cfd4d53ac6e3e0cd9bfe0df P b81c4e7f2fd64049a293624f0e0911f6: Log is configured to *not* fsync() on all Append() calls
I20260811 22:25:50.774595 7360 tablet_bootstrap.cc:492] T c84d30630cfd4d53ac6e3e0cd9bfe0df P 46d01e363bda4bef9d9949821d670926: Bootstrap starting.
I20260811 22:25:50.776399 7359 tablet_bootstrap.cc:654] T c84d30630cfd4d53ac6e3e0cd9bfe0df P f83b5295dfc64676b328137aa7d59990: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:50.777688 7360 tablet_bootstrap.cc:654] T c84d30630cfd4d53ac6e3e0cd9bfe0df P 46d01e363bda4bef9d9949821d670926: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:50.777650 7359 log.cc:824] T c84d30630cfd4d53ac6e3e0cd9bfe0df P f83b5295dfc64676b328137aa7d59990: Log is configured to *not* fsync() on all Append() calls
I20260811 22:25:50.778772 7360 log.cc:824] T c84d30630cfd4d53ac6e3e0cd9bfe0df P 46d01e363bda4bef9d9949821d670926: Log is configured to *not* fsync() on all Append() calls
I20260811 22:25:50.780619 7359 tablet_bootstrap.cc:492] T c84d30630cfd4d53ac6e3e0cd9bfe0df P f83b5295dfc64676b328137aa7d59990: No bootstrap required, opened a new log
I20260811 22:25:50.780731 7358 tablet_bootstrap.cc:492] T c84d30630cfd4d53ac6e3e0cd9bfe0df P b81c4e7f2fd64049a293624f0e0911f6: No bootstrap required, opened a new log
I20260811 22:25:50.780869 7359 ts_tablet_manager.cc:1397] T c84d30630cfd4d53ac6e3e0cd9bfe0df P f83b5295dfc64676b328137aa7d59990: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260811 22:25:50.781005 7358 ts_tablet_manager.cc:1397] T c84d30630cfd4d53ac6e3e0cd9bfe0df P b81c4e7f2fd64049a293624f0e0911f6: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.009s
I20260811 22:25:50.781617 7360 tablet_bootstrap.cc:492] T c84d30630cfd4d53ac6e3e0cd9bfe0df P 46d01e363bda4bef9d9949821d670926: No bootstrap required, opened a new log
I20260811 22:25:50.781867 7360 ts_tablet_manager.cc:1397] T c84d30630cfd4d53ac6e3e0cd9bfe0df P 46d01e363bda4bef9d9949821d670926: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260811 22:25:50.788159 7359 raft_consensus.cc:348] T c84d30630cfd4d53ac6e3e0cd9bfe0df P f83b5295dfc64676b328137aa7d59990 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81c4e7f2fd64049a293624f0e0911f6" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 32951 } } peers { permanent_uuid: "f83b5295dfc64676b328137aa7d59990" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43483 } } peers { permanent_uuid: "46d01e363bda4bef9d9949821d670926" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38165 } }
I20260811 22:25:50.788486 7359 raft_consensus.cc:374] T c84d30630cfd4d53ac6e3e0cd9bfe0df P f83b5295dfc64676b328137aa7d59990 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:25:50.788580 7359 raft_consensus.cc:729] T c84d30630cfd4d53ac6e3e0cd9bfe0df P f83b5295dfc64676b328137aa7d59990 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83b5295dfc64676b328137aa7d59990, State: Initialized, Role: FOLLOWER
I20260811 22:25:50.789110 7359 consensus_queue.cc:260] T c84d30630cfd4d53ac6e3e0cd9bfe0df P f83b5295dfc64676b328137aa7d59990 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81c4e7f2fd64049a293624f0e0911f6" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 32951 } } peers { permanent_uuid: "f83b5295dfc64676b328137aa7d59990" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43483 } } peers { permanent_uuid: "46d01e363bda4bef9d9949821d670926" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38165 } }
I20260811 22:25:50.790083 6831 heartbeater.cc:503] Master 127.29.106.62:46681 was elected leader, sending a full tablet report...
I20260811 22:25:50.790669 7359 ts_tablet_manager.cc:1428] T c84d30630cfd4d53ac6e3e0cd9bfe0df P f83b5295dfc64676b328137aa7d59990: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260811 22:25:50.791239 7358 raft_consensus.cc:348] T c84d30630cfd4d53ac6e3e0cd9bfe0df P b81c4e7f2fd64049a293624f0e0911f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81c4e7f2fd64049a293624f0e0911f6" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 32951 } } peers { permanent_uuid: "f83b5295dfc64676b328137aa7d59990" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43483 } } peers { permanent_uuid: "46d01e363bda4bef9d9949821d670926" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38165 } }
I20260811 22:25:50.791707 7358 raft_consensus.cc:374] T c84d30630cfd4d53ac6e3e0cd9bfe0df P b81c4e7f2fd64049a293624f0e0911f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:25:50.791814 7358 raft_consensus.cc:729] T c84d30630cfd4d53ac6e3e0cd9bfe0df P b81c4e7f2fd64049a293624f0e0911f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b81c4e7f2fd64049a293624f0e0911f6, State: Initialized, Role: FOLLOWER
I20260811 22:25:50.791746 7360 raft_consensus.cc:348] T c84d30630cfd4d53ac6e3e0cd9bfe0df P 46d01e363bda4bef9d9949821d670926 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81c4e7f2fd64049a293624f0e0911f6" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 32951 } } peers { permanent_uuid: "f83b5295dfc64676b328137aa7d59990" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43483 } } peers { permanent_uuid: "46d01e363bda4bef9d9949821d670926" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38165 } }
I20260811 22:25:50.792205 7360 raft_consensus.cc:374] T c84d30630cfd4d53ac6e3e0cd9bfe0df P 46d01e363bda4bef9d9949821d670926 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:25:50.792307 7360 raft_consensus.cc:729] T c84d30630cfd4d53ac6e3e0cd9bfe0df P 46d01e363bda4bef9d9949821d670926 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46d01e363bda4bef9d9949821d670926, State: Initialized, Role: FOLLOWER
I20260811 22:25:50.792379 7358 consensus_queue.cc:260] T c84d30630cfd4d53ac6e3e0cd9bfe0df P b81c4e7f2fd64049a293624f0e0911f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81c4e7f2fd64049a293624f0e0911f6" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 32951 } } peers { permanent_uuid: "f83b5295dfc64676b328137aa7d59990" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43483 } } peers { permanent_uuid: "46d01e363bda4bef9d9949821d670926" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38165 } }
I20260811 22:25:50.792862 7360 consensus_queue.cc:260] T c84d30630cfd4d53ac6e3e0cd9bfe0df P 46d01e363bda4bef9d9949821d670926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81c4e7f2fd64049a293624f0e0911f6" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 32951 } } peers { permanent_uuid: "f83b5295dfc64676b328137aa7d59990" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43483 } } peers { permanent_uuid: "46d01e363bda4bef9d9949821d670926" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38165 } }
I20260811 22:25:50.793413 7079 heartbeater.cc:503] Master 127.29.106.62:46681 was elected leader, sending a full tablet report...
I20260811 22:25:50.794449 7358 ts_tablet_manager.cc:1428] T c84d30630cfd4d53ac6e3e0cd9bfe0df P b81c4e7f2fd64049a293624f0e0911f6: Time spent starting tablet: real 0.013s user 0.004s sys 0.010s
I20260811 22:25:50.796074 7322 heartbeater.cc:503] Master 127.29.106.62:46681 was elected leader, sending a full tablet report...
I20260811 22:25:50.796646 7360 ts_tablet_manager.cc:1428] T c84d30630cfd4d53ac6e3e0cd9bfe0df P 46d01e363bda4bef9d9949821d670926: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
I20260811 22:25:50.848539 7372 raft_consensus.cc:482] T c84d30630cfd4d53ac6e3e0cd9bfe0df P f83b5295dfc64676b328137aa7d59990 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:25:50.848815 7372 raft_consensus.cc:504] T c84d30630cfd4d53ac6e3e0cd9bfe0df P f83b5295dfc64676b328137aa7d59990 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81c4e7f2fd64049a293624f0e0911f6" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 32951 } } peers { permanent_uuid: "f83b5295dfc64676b328137aa7d59990" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43483 } } peers { permanent_uuid: "46d01e363bda4bef9d9949821d670926" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38165 } }
I20260811 22:25:50.850432 7372 leader_election.cc:288] T c84d30630cfd4d53ac6e3e0cd9bfe0df P f83b5295dfc64676b328137aa7d59990 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b81c4e7f2fd64049a293624f0e0911f6 (127.29.106.2:32951), 46d01e363bda4bef9d9949821d670926 (127.29.106.3:38165)
I20260811 22:25:50.853415 7356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:50.851022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:53115 (local address 127.29.106.3:38165)
0811 22:25:50.851202 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:50.851208 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:50.851569 (+ 361us) server_negotiation.cc:407] Connection header received
0811 22:25:50.851734 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:50.851738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:50.851790 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:50.851893 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:50.852822 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:50.852828 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:50.852833 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:25:50.852852 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:50.852894 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:25:50.852911 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:50.853040 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:50.853109 (+ 69us) server_negotiation.cc:299] Negotiation successful
0811 22:25:50.853168 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260811 22:25:50.853415 7348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:50.850724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:52045 (local address 127.29.106.2:32951)
0811 22:25:50.850900 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:50.850906 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:50.851818 (+ 912us) server_negotiation.cc:407] Connection header received
0811 22:25:50.851919 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:50.851924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:50.851981 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:50.852067 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:50.852869 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:50.852874 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:50.852878 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:25:50.852894 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:50.852927 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:25:50.852941 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:50.853044 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:50.853111 (+ 67us) server_negotiation.cc:299] Negotiation successful
0811 22:25:50.853168 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260811 22:25:50.853754 7381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:50.850681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.3:38165 (local address 127.29.106.1:53115)
0811 22:25:50.851460 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:50.851478 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:50.851612 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:50.851941 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:50.851944 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:50.851964 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:50.852597 (+ 633us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 22:25:50.852601 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:50.852619 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 22:25:50.852674 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:50.853006 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:50.853011 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:50.853015 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 22:25:50.853170 (+ 155us) client_negotiation.cc:240] Negotiation successful
0811 22:25:50.853249 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"mutex_wait_us":21,"thread_start_us":420,"threads_started":1}
I20260811 22:25:50.853825 7383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:50.850760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.2:32951 (local address 127.29.106.1:52045)
0811 22:25:50.851722 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:50.851735 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:50.851823 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:50.852109 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:50.852113 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:50.852131 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:50.852692 (+ 561us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 22:25:50.852694 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:50.852707 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 22:25:50.852740 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:50.853019 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:50.853023 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:50.853027 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 22:25:50.853180 (+ 153us) client_negotiation.cc:240] Negotiation successful
0811 22:25:50.853234 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":865,"thread_start_us":653,"threads_started":1}
I20260811 22:25:50.854624 6988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c84d30630cfd4d53ac6e3e0cd9bfe0df" candidate_uuid: "f83b5295dfc64676b328137aa7d59990" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b81c4e7f2fd64049a293624f0e0911f6" is_pre_election: true
I20260811 22:25:50.854682 7242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c84d30630cfd4d53ac6e3e0cd9bfe0df" candidate_uuid: "f83b5295dfc64676b328137aa7d59990" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46d01e363bda4bef9d9949821d670926" is_pre_election: true
I20260811 22:25:50.855034 6988 raft_consensus.cc:2454] T c84d30630cfd4d53ac6e3e0cd9bfe0df P b81c4e7f2fd64049a293624f0e0911f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f83b5295dfc64676b328137aa7d59990 in term 0.
I20260811 22:25:50.855127 7242 raft_consensus.cc:2454] T c84d30630cfd4d53ac6e3e0cd9bfe0df P 46d01e363bda4bef9d9949821d670926 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f83b5295dfc64676b328137aa7d59990 in term 0.
I20260811 22:25:50.855588 6613 leader_election.cc:302] T c84d30630cfd4d53ac6e3e0cd9bfe0df P f83b5295dfc64676b328137aa7d59990 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b81c4e7f2fd64049a293624f0e0911f6, f83b5295dfc64676b328137aa7d59990; no voters:
I20260811 22:25:50.856005 7372 raft_consensus.cc:2781] T c84d30630cfd4d53ac6e3e0cd9bfe0df P f83b5295dfc64676b328137aa7d59990 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:25:50.856171 7372 raft_consensus.cc:482] T c84d30630cfd4d53ac6e3e0cd9bfe0df P f83b5295dfc64676b328137aa7d59990 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:25:50.856285 7372 raft_consensus.cc:3037] T c84d30630cfd4d53ac6e3e0cd9bfe0df P f83b5295dfc64676b328137aa7d59990 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:50.858574 7372 raft_consensus.cc:504] T c84d30630cfd4d53ac6e3e0cd9bfe0df P f83b5295dfc64676b328137aa7d59990 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81c4e7f2fd64049a293624f0e0911f6" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 32951 } } peers { permanent_uuid: "f83b5295dfc64676b328137aa7d59990" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43483 } } peers { permanent_uuid: "46d01e363bda4bef9d9949821d670926" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38165 } }
I20260811 22:25:50.859145 7372 leader_election.cc:288] T c84d30630cfd4d53ac6e3e0cd9bfe0df P f83b5295dfc64676b328137aa7d59990 [CANDIDATE]: Term 1 election: Requested vote from peers b81c4e7f2fd64049a293624f0e0911f6 (127.29.106.2:32951), 46d01e363bda4bef9d9949821d670926 (127.29.106.3:38165)
I20260811 22:25:50.859529 6988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c84d30630cfd4d53ac6e3e0cd9bfe0df" candidate_uuid: "f83b5295dfc64676b328137aa7d59990" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b81c4e7f2fd64049a293624f0e0911f6"
I20260811 22:25:50.859661 7242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c84d30630cfd4d53ac6e3e0cd9bfe0df" candidate_uuid: "f83b5295dfc64676b328137aa7d59990" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46d01e363bda4bef9d9949821d670926"
I20260811 22:25:50.859719 6988 raft_consensus.cc:3037] T c84d30630cfd4d53ac6e3e0cd9bfe0df P b81c4e7f2fd64049a293624f0e0911f6 [term 0 FOLLOWER]: Advancing to term 1
W20260811 22:25:50.859740 6832 tablet.cc:2369] T c84d30630cfd4d53ac6e3e0cd9bfe0df P f83b5295dfc64676b328137aa7d59990: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:25:50.859843 7242 raft_consensus.cc:3037] T c84d30630cfd4d53ac6e3e0cd9bfe0df P 46d01e363bda4bef9d9949821d670926 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:50.862198 6988 raft_consensus.cc:2454] T c84d30630cfd4d53ac6e3e0cd9bfe0df P b81c4e7f2fd64049a293624f0e0911f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f83b5295dfc64676b328137aa7d59990 in term 1.
I20260811 22:25:50.862648 6613 leader_election.cc:302] T c84d30630cfd4d53ac6e3e0cd9bfe0df P f83b5295dfc64676b328137aa7d59990 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b81c4e7f2fd64049a293624f0e0911f6, f83b5295dfc64676b328137aa7d59990; no voters:
I20260811 22:25:50.862946 7372 raft_consensus.cc:2781] T c84d30630cfd4d53ac6e3e0cd9bfe0df P f83b5295dfc64676b328137aa7d59990 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:25:50.862979 7242 raft_consensus.cc:2454] T c84d30630cfd4d53ac6e3e0cd9bfe0df P 46d01e363bda4bef9d9949821d670926 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f83b5295dfc64676b328137aa7d59990 in term 1.
I20260811 22:25:50.866537 7372 raft_consensus.cc:686] T c84d30630cfd4d53ac6e3e0cd9bfe0df P f83b5295dfc64676b328137aa7d59990 [term 1 LEADER]: Becoming Leader. State: Replica: f83b5295dfc64676b328137aa7d59990, State: Running, Role: LEADER
I20260811 22:25:50.867071 7372 consensus_queue.cc:237] T c84d30630cfd4d53ac6e3e0cd9bfe0df P f83b5295dfc64676b328137aa7d59990 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81c4e7f2fd64049a293624f0e0911f6" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 32951 } } peers { permanent_uuid: "f83b5295dfc64676b328137aa7d59990" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43483 } } peers { permanent_uuid: "46d01e363bda4bef9d9949821d670926" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38165 } }
I20260811 22:25:50.871239 6456 catalog_manager.cc:5495] T c84d30630cfd4d53ac6e3e0cd9bfe0df P f83b5295dfc64676b328137aa7d59990 reported cstate change: term changed from 0 to 1, leader changed from <none> to f83b5295dfc64676b328137aa7d59990 (127.29.106.1). New cstate: current_term: 1 leader_uuid: "f83b5295dfc64676b328137aa7d59990" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81c4e7f2fd64049a293624f0e0911f6" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 32951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f83b5295dfc64676b328137aa7d59990" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 43483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46d01e363bda4bef9d9949821d670926" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38165 } health_report { overall_health: UNKNOWN } } }
I20260811 22:25:50.904848 7351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:50.902934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47222 (local address 127.29.106.1:43483)
0811 22:25:50.903091 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:50.903096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:50.903212 (+ 116us) server_negotiation.cc:407] Connection header received
0811 22:25:50.903416 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:50.903419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:50.903507 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:50.903613 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:50.904398 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:50.904401 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:50.904404 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:25:50.904415 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:50.904439 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:25:50.904450 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:50.904531 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:50.904594 (+ 63us) server_negotiation.cc:299] Negotiation successful
0811 22:25:50.904639 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260811 22:25:50.912541 7242 raft_consensus.cc:1261] T c84d30630cfd4d53ac6e3e0cd9bfe0df P 46d01e363bda4bef9d9949821d670926 [term 1 FOLLOWER]: Refusing update from remote peer f83b5295dfc64676b328137aa7d59990: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:25:50.912554 6988 raft_consensus.cc:1261] T c84d30630cfd4d53ac6e3e0cd9bfe0df P b81c4e7f2fd64049a293624f0e0911f6 [term 1 FOLLOWER]: Refusing update from remote peer f83b5295dfc64676b328137aa7d59990: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:25:50.913399 7372 consensus_queue.cc:1035] T c84d30630cfd4d53ac6e3e0cd9bfe0df P f83b5295dfc64676b328137aa7d59990 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46d01e363bda4bef9d9949821d670926" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:25:50.913705 7384 consensus_queue.cc:1035] T c84d30630cfd4d53ac6e3e0cd9bfe0df P f83b5295dfc64676b328137aa7d59990 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b81c4e7f2fd64049a293624f0e0911f6" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 32951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:25:50.929975 7389 mvcc.cc:205] Tried to move back new op lower bound from 7317451370128629760 to 7317451369955393536. Current Snapshot: MvccSnapshot[applied={T|T < 7317451370128629760}]
I20260811 22:25:50.941815 6456 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:25:50.942067 6456 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:25:50.945468 6456 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2af2db0d30de47208149ffda390a641a: Sending DeleteTablet for 3 replicas of tablet c84d30630cfd4d53ac6e3e0cd9bfe0df
I20260811 22:25:50.946455 6697 tablet_service.cc:1514] Processing DeleteTablet for tablet c84d30630cfd4d53ac6e3e0cd9bfe0df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:25:50 UTC) from {username='slave'} at 127.0.0.1:47214
I20260811 22:25:50.946455 7210 tablet_service.cc:1514] Processing DeleteTablet for tablet c84d30630cfd4d53ac6e3e0cd9bfe0df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:25:50 UTC) from {username='slave'} at 127.0.0.1:40754
I20260811 22:25:50.947281 7408 tablet_replica.cc:330] T c84d30630cfd4d53ac6e3e0cd9bfe0df P f83b5295dfc64676b328137aa7d59990: stopping tablet replica
I20260811 22:25:50.947288 7407 tablet_replica.cc:330] T c84d30630cfd4d53ac6e3e0cd9bfe0df P 46d01e363bda4bef9d9949821d670926: stopping tablet replica
I20260811 22:25:50.947048 6948 tablet_service.cc:1514] Processing DeleteTablet for tablet c84d30630cfd4d53ac6e3e0cd9bfe0df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:25:50 UTC) from {username='slave'} at 127.0.0.1:49176
I20260811 22:25:50.947749 7408 raft_consensus.cc:2229] T c84d30630cfd4d53ac6e3e0cd9bfe0df P f83b5295dfc64676b328137aa7d59990 [term 1 LEADER]: Raft consensus shutting down.
I20260811 22:25:50.948467 7408 raft_consensus.cc:2258] T c84d30630cfd4d53ac6e3e0cd9bfe0df P f83b5295dfc64676b328137aa7d59990 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:25:50.948706 7407 raft_consensus.cc:2229] T c84d30630cfd4d53ac6e3e0cd9bfe0df P 46d01e363bda4bef9d9949821d670926 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:25:50.949198 7407 raft_consensus.cc:2258] T c84d30630cfd4d53ac6e3e0cd9bfe0df P 46d01e363bda4bef9d9949821d670926 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:25:50.949697 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 6578
I20260811 22:25:50.950971 7408 ts_tablet_manager.cc:1905] T c84d30630cfd4d53ac6e3e0cd9bfe0df P f83b5295dfc64676b328137aa7d59990: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:25:50.958297 7408 ts_tablet_manager.cc:1918] T c84d30630cfd4d53ac6e3e0cd9bfe0df P f83b5295dfc64676b328137aa7d59990: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:25:50.958478 7408 log.cc:1196] T c84d30630cfd4d53ac6e3e0cd9bfe0df P f83b5295dfc64676b328137aa7d59990: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-0/wal/wals/c84d30630cfd4d53ac6e3e0cd9bfe0df
I20260811 22:25:50.959074 7408 ts_tablet_manager.cc:1939] T c84d30630cfd4d53ac6e3e0cd9bfe0df P f83b5295dfc64676b328137aa7d59990: Deleting consensus metadata
I20260811 22:25:50.959476 7407 ts_tablet_manager.cc:1905] T c84d30630cfd4d53ac6e3e0cd9bfe0df P 46d01e363bda4bef9d9949821d670926: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:25:50.960827 6432 catalog_manager.cc:4841] TS f83b5295dfc64676b328137aa7d59990 (127.29.106.1:43483): tablet c84d30630cfd4d53ac6e3e0cd9bfe0df (table test-table [id=3640a4e64a5e4522928c32067e8d7509]) successfully deleted
I20260811 22:25:50.963806 7409 tablet_replica.cc:330] T c84d30630cfd4d53ac6e3e0cd9bfe0df P b81c4e7f2fd64049a293624f0e0911f6: stopping tablet replica
I20260811 22:25:50.964234 7409 raft_consensus.cc:2229] T c84d30630cfd4d53ac6e3e0cd9bfe0df P b81c4e7f2fd64049a293624f0e0911f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:25:50.964740 7409 raft_consensus.cc:2258] T c84d30630cfd4d53ac6e3e0cd9bfe0df P b81c4e7f2fd64049a293624f0e0911f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:25:50.966951 7407 ts_tablet_manager.cc:1918] T c84d30630cfd4d53ac6e3e0cd9bfe0df P 46d01e363bda4bef9d9949821d670926: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:25:50.967108 7407 log.cc:1196] T c84d30630cfd4d53ac6e3e0cd9bfe0df P 46d01e363bda4bef9d9949821d670926: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-2/wal/wals/c84d30630cfd4d53ac6e3e0cd9bfe0df
I20260811 22:25:50.967587 7409 ts_tablet_manager.cc:1905] T c84d30630cfd4d53ac6e3e0cd9bfe0df P b81c4e7f2fd64049a293624f0e0911f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:25:50.967674 7407 ts_tablet_manager.cc:1939] T c84d30630cfd4d53ac6e3e0cd9bfe0df P 46d01e363bda4bef9d9949821d670926: Deleting consensus metadata
I20260811 22:25:50.968200 7351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:50.950835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47230 (local address 127.29.106.1:43483)
0811 22:25:50.951019 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:50.951024 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:50.965547 (+ 14523us) server_negotiation.cc:407] Connection header received
0811 22:25:50.965771 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:50.965776 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:50.965843 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:50.965955 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:50.967175 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:50.967181 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:50.967186 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:25:50.967207 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:50.967250 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:25:50.967357 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:50.967461 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:50.967837 (+ 376us) server_negotiation.cc:299] Negotiation successful
0811 22:25:50.967908 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260811 22:25:50.969383 6823 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:25:50.969475 6433 catalog_manager.cc:4841] TS 46d01e363bda4bef9d9949821d670926 (127.29.106.3:38165): tablet c84d30630cfd4d53ac6e3e0cd9bfe0df (table test-table [id=3640a4e64a5e4522928c32067e8d7509]) successfully deleted
I20260811 22:25:50.974267 7409 ts_tablet_manager.cc:1918] T c84d30630cfd4d53ac6e3e0cd9bfe0df P b81c4e7f2fd64049a293624f0e0911f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:25:50.974426 7409 log.cc:1196] T c84d30630cfd4d53ac6e3e0cd9bfe0df P b81c4e7f2fd64049a293624f0e0911f6: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786487105757681-30120-0/minicluster-data/ts-1/wal/wals/c84d30630cfd4d53ac6e3e0cd9bfe0df
I20260811 22:25:50.974953 7409 ts_tablet_manager.cc:1939] T c84d30630cfd4d53ac6e3e0cd9bfe0df P b81c4e7f2fd64049a293624f0e0911f6: Deleting consensus metadata
I20260811 22:25:50.976703 6437 catalog_manager.cc:4841] TS b81c4e7f2fd64049a293624f0e0911f6 (127.29.106.2:32951): tablet c84d30630cfd4d53ac6e3e0cd9bfe0df (table test-table [id=3640a4e64a5e4522928c32067e8d7509]) successfully deleted
I20260811 22:25:51.062547 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 6578
I20260811 22:25:51.083825 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 6835
I20260811 22:25:51.086259 7348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:51.084507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49180 (local address 127.29.106.2:32951)
0811 22:25:51.084662 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:51.084666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:51.084698 (+ 32us) server_negotiation.cc:407] Connection header received
0811 22:25:51.084847 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:51.084851 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:51.084938 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:51.085020 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:51.085796 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:51.085801 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:51.085804 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:25:51.085820 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:51.085850 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:25:51.085862 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:51.085962 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:51.085996 (+ 34us) server_negotiation.cc:299] Negotiation successful
0811 22:25:51.086043 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260811 22:25:51.086838 7071 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:25:51.172335 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 6835
I20260811 22:25:51.191329 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 7085
I20260811 22:25:51.194068 7356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:51.192160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40764 (local address 127.29.106.3:38165)
0811 22:25:51.192321 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:51.192326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:51.192345 (+ 19us) server_negotiation.cc:407] Connection header received
0811 22:25:51.192498 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:51.192502 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:51.192583 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:51.192663 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:51.193501 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:51.193507 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:51.193510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:25:51.193527 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:51.193572 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:25:51.193585 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:51.193696 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:51.193732 (+ 36us) server_negotiation.cc:299] Negotiation successful
0811 22:25:51.193779 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260811 22:25:51.194715 7316 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:25:51.286935 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 7085
I20260811 22:25:51.309881 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 6404
I20260811 22:25:51.310956 6513 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:25:51.414891 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 6404
2026-08-11T22:25:51Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2555 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260811 22:25:51.464641 30120 mini_oidc.cc:152] Starting JWKS server
I20260811 22:25:51.467604 30120 webserver.cc:460] Webserver started at https://127.0.0.1:38849/ using document root <none> and password file <none>
I20260811 22:25:51.467943 30120 mini_oidc.cc:189] Starting OIDC Discovery server
I20260811 22:25:51.468575 30120 webserver.cc:460] Webserver started at http://127.0.0.1:39137/ using document root <none> and password file <none>
2026-08-11T22:25:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:25:51Z Disabled control of system clock
I20260811 22:25:51.497975 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:32905
--webserver_interface=127.29.106.62
--webserver_port=0
--builtin_ntp_servers=127.29.106.20:45201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:32905
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38849/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786487105757681-30120-0/testchainca.cert with env {}
W20260811 22:25:51.814249 7429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:51.814508 7429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:51.814605 7429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:51.821477 7429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:25:51.821566 7429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:51.821611 7429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:51.821671 7429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:25:51.821714 7429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260811 22:25:51.821774 7429 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260811 22:25:51.821812 7429 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38849/jwks.json
I20260811 22:25:51.831128 7429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:45201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:32905
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:32905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38849/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.106.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786487105757681-30120-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:25:51.832227 7429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:51.833367 7429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:51.844187 7441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:51.844213 7442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:51.844120 7445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:51.845098 7429 server_base.cc:1034] running on GCE node
I20260811 22:25:51.845781 7429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:51.846953 7429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:51.848155 7429 hybrid_clock.cc:648] HybridClock initialized: now 1786487151848109 us; error 31 us; skew 500 ppm
I20260811 22:25:51.848464 7429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 22:25:51.850489 7429 webserver.cc:460] Webserver started at http://127.29.106.62:42943/ using document root <none> and password file <none>
I20260811 22:25:51.851092 7429 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:51.851190 7429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:51.851497 7429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:51.854203 7429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "daf742ada9eb4144bfa0b04406ddda98"
format_stamp: "Formatted at 2026-08-11 22:25:51 on dist-test-slave-gjrz"
server_key: "cf7f6978f5db3c7405e6194ca7e4f6b5"
server_key_iv: "ad84d184b168642692a51dffe933fa87"
server_key_version: "encryptionkey@0"
I20260811 22:25:51.854955 7429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "daf742ada9eb4144bfa0b04406ddda98"
format_stamp: "Formatted at 2026-08-11 22:25:51 on dist-test-slave-gjrz"
server_key: "cf7f6978f5db3c7405e6194ca7e4f6b5"
server_key_iv: "ad84d184b168642692a51dffe933fa87"
server_key_version: "encryptionkey@0"
I20260811 22:25:51.859627 7429 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260811 22:25:51.862767 7451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:51.864053 7429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:25:51.864212 7429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "daf742ada9eb4144bfa0b04406ddda98"
format_stamp: "Formatted at 2026-08-11 22:25:51 on dist-test-slave-gjrz"
server_key: "cf7f6978f5db3c7405e6194ca7e4f6b5"
server_key_iv: "ad84d184b168642692a51dffe933fa87"
server_key_version: "encryptionkey@0"
I20260811 22:25:51.864403 7429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:51.876896 7429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:51.877592 7429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:51.877852 7429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:51.894470 7429 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.62:32905
I20260811 22:25:51.894490 7573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.62:32905 every 8 connection(s)
I20260811 22:25:51.895697 7429 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:25:51.899148 7575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:51.903690 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 7429
I20260811 22:25:51.903853 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:25:51.904449 30120 external_mini_cluster.cc:1428] Setting key e5554352dff1165e2fcc33668dcedc9f
I20260811 22:25:51.909710 7575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daf742ada9eb4144bfa0b04406ddda98: Bootstrap starting.
I20260811 22:25:51.913138 7575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P daf742ada9eb4144bfa0b04406ddda98: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:51.914259 7575 log.cc:824] T 00000000000000000000000000000000 P daf742ada9eb4144bfa0b04406ddda98: Log is configured to *not* fsync() on all Append() calls
I20260811 22:25:51.917189 7575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daf742ada9eb4144bfa0b04406ddda98: No bootstrap required, opened a new log
I20260811 22:25:51.917780 7581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:51.906147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55728 (local address 127.29.106.62:32905)
0811 22:25:51.907110 (+ 963us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:51.907120 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:51.907164 (+ 44us) server_negotiation.cc:407] Connection header received
0811 22:25:51.908866 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:51.908890 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:51.909309 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:51.909610 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:51.910190 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:51.911635 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:51.913478 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:51.914147 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:51.914296 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:51.915594 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:51.915603 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:51.915618 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:25:51.916165 (+ 547us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:51.916241 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:25:51.916263 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:51.916348 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:51.916573 (+ 225us) server_negotiation.cc:299] Negotiation successful
0811 22:25:51.916852 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":334,"threads_started":1}
I20260811 22:25:51.921336 30120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:25:51.925685 7575 raft_consensus.cc:348] T 00000000000000000000000000000000 P daf742ada9eb4144bfa0b04406ddda98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daf742ada9eb4144bfa0b04406ddda98" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 32905 } }
I20260811 22:25:51.925978 7575 raft_consensus.cc:374] T 00000000000000000000000000000000 P daf742ada9eb4144bfa0b04406ddda98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:25:51.926056 7575 raft_consensus.cc:729] T 00000000000000000000000000000000 P daf742ada9eb4144bfa0b04406ddda98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daf742ada9eb4144bfa0b04406ddda98, State: Initialized, Role: FOLLOWER
I20260811 22:25:51.926512 7575 consensus_queue.cc:260] T 00000000000000000000000000000000 P daf742ada9eb4144bfa0b04406ddda98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daf742ada9eb4144bfa0b04406ddda98" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 32905 } }
I20260811 22:25:51.926707 7575 raft_consensus.cc:388] T 00000000000000000000000000000000 P daf742ada9eb4144bfa0b04406ddda98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:25:51.926798 7575 raft_consensus.cc:482] T 00000000000000000000000000000000 P daf742ada9eb4144bfa0b04406ddda98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:25:51.926920 7575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P daf742ada9eb4144bfa0b04406ddda98 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:51.928920 7575 raft_consensus.cc:504] T 00000000000000000000000000000000 P daf742ada9eb4144bfa0b04406ddda98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daf742ada9eb4144bfa0b04406ddda98" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 32905 } }
I20260811 22:25:51.929345 7575 leader_election.cc:302] T 00000000000000000000000000000000 P daf742ada9eb4144bfa0b04406ddda98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: daf742ada9eb4144bfa0b04406ddda98; no voters:
I20260811 22:25:51.929840 7575 leader_election.cc:288] T 00000000000000000000000000000000 P daf742ada9eb4144bfa0b04406ddda98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:25:51.929965 7590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P daf742ada9eb4144bfa0b04406ddda98 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:25:51.930658 7590 raft_consensus.cc:686] T 00000000000000000000000000000000 P daf742ada9eb4144bfa0b04406ddda98 [term 1 LEADER]: Becoming Leader. State: Replica: daf742ada9eb4144bfa0b04406ddda98, State: Running, Role: LEADER
I20260811 22:25:51.931185 7590 consensus_queue.cc:237] T 00000000000000000000000000000000 P daf742ada9eb4144bfa0b04406ddda98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daf742ada9eb4144bfa0b04406ddda98" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 32905 } }
I20260811 22:25:51.931815 7575 sys_catalog.cc:564] T 00000000000000000000000000000000 P daf742ada9eb4144bfa0b04406ddda98 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:25:51.933854 7595 sys_catalog.cc:455] T 00000000000000000000000000000000 P daf742ada9eb4144bfa0b04406ddda98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader daf742ada9eb4144bfa0b04406ddda98. Latest consensus state: current_term: 1 leader_uuid: "daf742ada9eb4144bfa0b04406ddda98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daf742ada9eb4144bfa0b04406ddda98" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 32905 } } }
I20260811 22:25:51.934350 7595 sys_catalog.cc:458] T 00000000000000000000000000000000 P daf742ada9eb4144bfa0b04406ddda98 [sys.catalog]: This master's current role is: LEADER
I20260811 22:25:51.934187 7591 sys_catalog.cc:455] T 00000000000000000000000000000000 P daf742ada9eb4144bfa0b04406ddda98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "daf742ada9eb4144bfa0b04406ddda98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daf742ada9eb4144bfa0b04406ddda98" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 32905 } } }
I20260811 22:25:51.934835 7591 sys_catalog.cc:458] T 00000000000000000000000000000000 P daf742ada9eb4144bfa0b04406ddda98 [sys.catalog]: This master's current role is: LEADER
I20260811 22:25:51.936444 7606 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:25:51.940663 7606 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:25:51.948182 7606 catalog_manager.cc:1347] Generated new cluster ID: 2742ef7308d14594a018ee71dfa6e7a4
I20260811 22:25:51.948282 7606 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:25:51.962986 7606 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:25:51.964632 7606 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:25:51.975920 7606 catalog_manager.cc:5868] T 00000000000000000000000000000000 P daf742ada9eb4144bfa0b04406ddda98: Generated new TSK 0
I20260811 22:25:51.976799 7606 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260811 22:25:52.031028 7633 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260811 22:25:52.031225 7633 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260811 22:25:52.032657 7581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:52.003692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55738 (local address 127.29.106.62:32905)
0811 22:25:52.003910 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:52.003917 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:52.004345 (+ 428us) server_negotiation.cc:407] Connection header received
0811 22:25:52.004712 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:52.004715 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:52.004779 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:52.004890 (+ 111us) server_negotiation.cc:226] Negotiated authn=JWT
0811 22:25:52.005506 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:52.006224 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:52.008296 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:52.008748 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:52.008842 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:52.009432 (+ 590us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0811 22:25:52.031935 (+ 22503us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38849/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0811 22:25:52.032344 (+ 409us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55738: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38849/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":64}
W20260811 22:25:52.032781 7581 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55738: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38849/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260811 22:25:52.034266 7632 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:25:52.003600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:32905 (local address 127.0.0.1:55738)
0811 22:25:52.004241 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:52.004260 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:52.004450 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:52.004933 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:52.004939 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:52.004952 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0811 22:25:52.005366 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:52.005379 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:52.006350 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:52.006356 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:52.008160 (+ 1804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:52.008177 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:52.008918 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:52.008925 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:52.009123 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:52.009199 (+ 76us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0811 22:25:52.032208 (+ 23009us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38849/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0811 22:25:52.032516 (+ 308us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.29.106.62:32905: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38849/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":296,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.29.106.62:32905: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38849/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260811 22:25:52.036037 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 7429
I20260811 22:25:52.037160 7559 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:25:52.130601 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 7429
2026-08-11T22:25:52Z chronyd exiting
I20260811 22:25:52.187254 30120 test_util.cc:175] -----------------------------------------------
I20260811 22:25:52.187414 30120 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786487105757681-30120-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (727 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260811 22:25:52.191159 30120 mini_oidc.cc:152] Starting JWKS server
I20260811 22:25:52.192975 30120 webserver.cc:460] Webserver started at https://127.0.0.1:38907/ using document root <none> and password file <none>
I20260811 22:25:52.193189 30120 mini_oidc.cc:189] Starting OIDC Discovery server
I20260811 22:25:52.193818 30120 webserver.cc:460] Webserver started at http://127.0.0.1:37291/ using document root <none> and password file <none>
2026-08-11T22:25:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:25:52Z Disabled control of system clock
I20260811 22:25:52.222908 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:39321
--webserver_interface=127.29.106.62
--webserver_port=0
--builtin_ntp_servers=127.29.106.20:36515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:39321
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38907/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786487105757681-30120-0/testchainca.cert with env {}
W20260811 22:25:52.539026 7654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:52.539330 7654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:52.539474 7654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:52.546424 7654 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:25:52.546530 7654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:52.546600 7654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:52.546677 7654 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:25:52.546738 7654 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260811 22:25:52.546821 7654 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260811 22:25:52.546875 7654 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38907/jwks.json
I20260811 22:25:52.556257 7654 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:36515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:39321
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:39321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38907/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.106.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786487105757681-30120-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:25:52.557395 7654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:52.558627 7654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:52.569638 7662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:52.569638 7664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:52.569672 7666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:52.570482 7654 server_base.cc:1034] running on GCE node
I20260811 22:25:52.571210 7654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:52.572516 7654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:52.573725 7654 hybrid_clock.cc:648] HybridClock initialized: now 1786487152573680 us; error 35 us; skew 500 ppm
I20260811 22:25:52.574069 7654 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 22:25:52.576254 7654 webserver.cc:460] Webserver started at http://127.29.106.62:38267/ using document root <none> and password file <none>
I20260811 22:25:52.576933 7654 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:52.577039 7654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:52.577344 7654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:52.580145 7654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "7e4044bfc5f04809a96810cc279ef08d"
format_stamp: "Formatted at 2026-08-11 22:25:52 on dist-test-slave-gjrz"
server_key: "ee8ca28c17498b82cf0f3f4dce32c078"
server_key_iv: "eab64a280d3e4e9356952aa2364dac30"
server_key_version: "encryptionkey@0"
I20260811 22:25:52.580909 7654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "7e4044bfc5f04809a96810cc279ef08d"
format_stamp: "Formatted at 2026-08-11 22:25:52 on dist-test-slave-gjrz"
server_key: "ee8ca28c17498b82cf0f3f4dce32c078"
server_key_iv: "eab64a280d3e4e9356952aa2364dac30"
server_key_version: "encryptionkey@0"
I20260811 22:25:52.585834 7654 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260811 22:25:52.589107 7679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:52.590269 7654 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 22:25:52.590425 7654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "7e4044bfc5f04809a96810cc279ef08d"
format_stamp: "Formatted at 2026-08-11 22:25:52 on dist-test-slave-gjrz"
server_key: "ee8ca28c17498b82cf0f3f4dce32c078"
server_key_iv: "eab64a280d3e4e9356952aa2364dac30"
server_key_version: "encryptionkey@0"
I20260811 22:25:52.590620 7654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:52.605875 7654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:52.606563 7654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:52.606835 7654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:52.623118 7654 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.62:39321
I20260811 22:25:52.623152 7785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.62:39321 every 8 connection(s)
I20260811 22:25:52.624308 7654 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:25:52.627707 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 7654
I20260811 22:25:52.627827 7788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:52.627898 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:25:52.628501 30120 external_mini_cluster.cc:1428] Setting key c4a688a63d63a1a8e5251567e418ea52
I20260811 22:25:52.641625 7788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e4044bfc5f04809a96810cc279ef08d: Bootstrap starting.
I20260811 22:25:52.643502 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:52.630217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51880 (local address 127.29.106.62:39321)
0811 22:25:52.631365 (+ 1148us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:52.631379 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:52.631444 (+ 65us) server_negotiation.cc:407] Connection header received
0811 22:25:52.634024 (+ 2580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:52.634063 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:52.634630 (+ 567us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:52.635068 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:52.635589 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:52.637430 (+ 1841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:52.638618 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:52.639698 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:52.639857 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:52.640847 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:52.640858 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:52.640874 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:25:52.641533 (+ 659us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:52.641636 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:25:52.641662 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:52.641776 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:52.641978 (+ 202us) server_negotiation.cc:299] Negotiation successful
0811 22:25:52.642287 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":353,"threads_started":1}
I20260811 22:25:52.645123 7788 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e4044bfc5f04809a96810cc279ef08d: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:52.646227 7788 log.cc:824] T 00000000000000000000000000000000 P 7e4044bfc5f04809a96810cc279ef08d: Log is configured to *not* fsync() on all Append() calls
I20260811 22:25:52.647085 30120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:25:52.648929 7788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e4044bfc5f04809a96810cc279ef08d: No bootstrap required, opened a new log
I20260811 22:25:52.656579 7788 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e4044bfc5f04809a96810cc279ef08d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e4044bfc5f04809a96810cc279ef08d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 39321 } }
I20260811 22:25:52.656903 7788 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e4044bfc5f04809a96810cc279ef08d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:25:52.656996 7788 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e4044bfc5f04809a96810cc279ef08d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e4044bfc5f04809a96810cc279ef08d, State: Initialized, Role: FOLLOWER
I20260811 22:25:52.657486 7788 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e4044bfc5f04809a96810cc279ef08d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e4044bfc5f04809a96810cc279ef08d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 39321 } }
I20260811 22:25:52.657713 7788 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e4044bfc5f04809a96810cc279ef08d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:25:52.657833 7788 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e4044bfc5f04809a96810cc279ef08d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:25:52.657984 7788 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e4044bfc5f04809a96810cc279ef08d [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:52.660226 7788 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e4044bfc5f04809a96810cc279ef08d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e4044bfc5f04809a96810cc279ef08d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 39321 } }
I20260811 22:25:52.660750 7788 leader_election.cc:302] T 00000000000000000000000000000000 P 7e4044bfc5f04809a96810cc279ef08d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e4044bfc5f04809a96810cc279ef08d; no voters:
I20260811 22:25:52.661258 7788 leader_election.cc:288] T 00000000000000000000000000000000 P 7e4044bfc5f04809a96810cc279ef08d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:25:52.661378 7796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e4044bfc5f04809a96810cc279ef08d [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:25:52.662015 7796 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e4044bfc5f04809a96810cc279ef08d [term 1 LEADER]: Becoming Leader. State: Replica: 7e4044bfc5f04809a96810cc279ef08d, State: Running, Role: LEADER
I20260811 22:25:52.662514 7796 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e4044bfc5f04809a96810cc279ef08d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e4044bfc5f04809a96810cc279ef08d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 39321 } }
I20260811 22:25:52.663074 7788 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e4044bfc5f04809a96810cc279ef08d [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:25:52.665087 7798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e4044bfc5f04809a96810cc279ef08d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e4044bfc5f04809a96810cc279ef08d. Latest consensus state: current_term: 1 leader_uuid: "7e4044bfc5f04809a96810cc279ef08d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e4044bfc5f04809a96810cc279ef08d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 39321 } } }
I20260811 22:25:52.665483 7798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e4044bfc5f04809a96810cc279ef08d [sys.catalog]: This master's current role is: LEADER
I20260811 22:25:52.665113 7797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e4044bfc5f04809a96810cc279ef08d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e4044bfc5f04809a96810cc279ef08d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e4044bfc5f04809a96810cc279ef08d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 39321 } } }
I20260811 22:25:52.667511 7797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e4044bfc5f04809a96810cc279ef08d [sys.catalog]: This master's current role is: LEADER
I20260811 22:25:52.668076 7807 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:25:52.673213 7807 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:25:52.682169 7807 catalog_manager.cc:1347] Generated new cluster ID: aade06090b574c75b5654946bb730864
I20260811 22:25:52.682365 7807 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:25:52.701300 7807 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:25:52.702654 7807 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:25:52.710942 7807 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e4044bfc5f04809a96810cc279ef08d: Generated new TSK 0
I20260811 22:25:52.711671 7807 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260811 22:25:52.740597 7837 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260811 22:25:52.740689 7837 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260811 22:25:52.741890 7836 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:25:52.722607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:39321 (local address 127.0.0.1:51894)
0811 22:25:52.723150 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:52.723168 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:52.723310 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:52.723719 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:52.723723 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:52.723732 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0811 22:25:52.723955 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:52.723964 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:52.725359 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:52.725362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:52.726408 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:52.726420 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:52.727382 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:52.727386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:52.727521 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:52.727551 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0811 22:25:52.741438 (+ 13887us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38907/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0811 22:25:52.741610 (+ 172us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.29.106.62:39321: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38907/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":295,"threads_started":1}
I20260811 22:25:52.741912 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:52.722736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51894 (local address 127.29.106.62:39321)
0811 22:25:52.722918 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:52.722924 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:52.723247 (+ 323us) server_negotiation.cc:407] Connection header received
0811 22:25:52.723467 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:52.723474 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:52.723574 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:52.723725 (+ 151us) server_negotiation.cc:226] Negotiated authn=JWT
0811 22:25:52.724133 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:52.725240 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:52.726543 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:52.727223 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:52.727373 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:52.727801 (+ 428us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0811 22:25:52.741257 (+ 13456us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38907/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0811 22:25:52.741594 (+ 337us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51894: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38907/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":59}
W20260811 22:25:52.742019 7794 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51894: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38907/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260811 22:25:52.742946 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 7654
I20260811 22:25:52.743953 7773 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:25:52.835081 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 7654
2026-08-11T22:25:52Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (699 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260811 22:25:52.890919 30120 mini_oidc.cc:152] Starting JWKS server
I20260811 22:25:52.892822 30120 webserver.cc:460] Webserver started at https://127.0.0.1:46109/ using document root <none> and password file <none>
I20260811 22:25:52.893031 30120 mini_oidc.cc:189] Starting OIDC Discovery server
I20260811 22:25:52.893630 30120 webserver.cc:460] Webserver started at http://127.0.0.1:45141/ using document root <none> and password file <none>
2026-08-11T22:25:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:25:52Z Disabled control of system clock
I20260811 22:25:52.925156 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:39509
--webserver_interface=127.29.106.62
--webserver_port=0
--builtin_ntp_servers=127.29.106.20:40731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:39509
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46109/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260811 22:25:53.241317 7852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:53.241607 7852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:53.241742 7852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:53.248838 7852 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:25:53.248950 7852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:53.249015 7852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:53.249100 7852 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:25:53.249161 7852 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260811 22:25:53.249245 7852 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260811 22:25:53.249298 7852 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46109/jwks.json
I20260811 22:25:53.258206 7852 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:40731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:39509
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:39509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46109/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.106.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:25:53.259454 7852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:53.261147 7852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:53.272526 7860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:53.272588 7864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:53.272617 7861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:53.273669 7852 server_base.cc:1034] running on GCE node
I20260811 22:25:53.274396 7852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:53.275746 7852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:53.276947 7852 hybrid_clock.cc:648] HybridClock initialized: now 1786487153276887 us; error 54 us; skew 500 ppm
I20260811 22:25:53.277261 7852 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 22:25:53.279500 7852 webserver.cc:460] Webserver started at http://127.29.106.62:33851/ using document root <none> and password file <none>
I20260811 22:25:53.280153 7852 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:53.280254 7852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:53.280562 7852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:53.283452 7852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "d2dfd0e53a9a430b87f772625366f102"
format_stamp: "Formatted at 2026-08-11 22:25:53 on dist-test-slave-gjrz"
server_key: "6d88de6f23156424b3350dcc12e04da8"
server_key_iv: "aafebb02dc55742a580362375b3d1750"
server_key_version: "encryptionkey@0"
I20260811 22:25:53.284241 7852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "d2dfd0e53a9a430b87f772625366f102"
format_stamp: "Formatted at 2026-08-11 22:25:53 on dist-test-slave-gjrz"
server_key: "6d88de6f23156424b3350dcc12e04da8"
server_key_iv: "aafebb02dc55742a580362375b3d1750"
server_key_version: "encryptionkey@0"
I20260811 22:25:53.289101 7852 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260811 22:25:53.292403 7876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:53.293722 7852 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 22:25:53.293880 7852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "d2dfd0e53a9a430b87f772625366f102"
format_stamp: "Formatted at 2026-08-11 22:25:53 on dist-test-slave-gjrz"
server_key: "6d88de6f23156424b3350dcc12e04da8"
server_key_iv: "aafebb02dc55742a580362375b3d1750"
server_key_version: "encryptionkey@0"
I20260811 22:25:53.294097 7852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:53.306030 7852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:53.306761 7852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:53.307060 7852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:53.324546 7852 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.62:39509
I20260811 22:25:53.324590 7969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.62:39509 every 8 connection(s)
I20260811 22:25:53.325830 7852 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:25:53.329360 7970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:53.332742 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 7852
I20260811 22:25:53.332934 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:25:53.333726 30120 external_mini_cluster.cc:1428] Setting key 47a2f445093f4e0e991f27e638ca6782
I20260811 22:25:53.341353 7970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2dfd0e53a9a430b87f772625366f102: Bootstrap starting.
I20260811 22:25:53.344333 7970 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2dfd0e53a9a430b87f772625366f102: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:53.345399 7970 log.cc:824] T 00000000000000000000000000000000 P d2dfd0e53a9a430b87f772625366f102: Log is configured to *not* fsync() on all Append() calls
I20260811 22:25:53.346271 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:53.335605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47868 (local address 127.29.106.62:39509)
0811 22:25:53.336502 (+ 897us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:53.336513 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:53.336559 (+ 46us) server_negotiation.cc:407] Connection header received
0811 22:25:53.338102 (+ 1543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:53.338123 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:53.338578 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:53.338941 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:53.339508 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:53.340738 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:53.342028 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:53.342690 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:53.342859 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:53.343909 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:53.343917 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:53.343930 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:25:53.344426 (+ 496us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:53.344555 (+ 129us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:25:53.344579 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:53.344697 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:53.344902 (+ 205us) server_negotiation.cc:299] Negotiation successful
0811 22:25:53.345177 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":312,"threads_started":1}
I20260811 22:25:53.348384 7970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2dfd0e53a9a430b87f772625366f102: No bootstrap required, opened a new log
I20260811 22:25:53.349897 30120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:25:53.355751 7970 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2dfd0e53a9a430b87f772625366f102 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2dfd0e53a9a430b87f772625366f102" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 39509 } }
I20260811 22:25:53.356172 7970 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2dfd0e53a9a430b87f772625366f102 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:25:53.356262 7970 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2dfd0e53a9a430b87f772625366f102 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2dfd0e53a9a430b87f772625366f102, State: Initialized, Role: FOLLOWER
I20260811 22:25:53.356768 7970 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2dfd0e53a9a430b87f772625366f102 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2dfd0e53a9a430b87f772625366f102" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 39509 } }
I20260811 22:25:53.356983 7970 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2dfd0e53a9a430b87f772625366f102 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:25:53.357110 7970 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2dfd0e53a9a430b87f772625366f102 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:25:53.357262 7970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2dfd0e53a9a430b87f772625366f102 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:53.359256 7970 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2dfd0e53a9a430b87f772625366f102 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2dfd0e53a9a430b87f772625366f102" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 39509 } }
I20260811 22:25:53.359758 7970 leader_election.cc:302] T 00000000000000000000000000000000 P d2dfd0e53a9a430b87f772625366f102 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2dfd0e53a9a430b87f772625366f102; no voters:
I20260811 22:25:53.360313 7970 leader_election.cc:288] T 00000000000000000000000000000000 P d2dfd0e53a9a430b87f772625366f102 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:25:53.360451 7983 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2dfd0e53a9a430b87f772625366f102 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:25:53.361500 7983 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2dfd0e53a9a430b87f772625366f102 [term 1 LEADER]: Becoming Leader. State: Replica: d2dfd0e53a9a430b87f772625366f102, State: Running, Role: LEADER
I20260811 22:25:53.361997 7983 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2dfd0e53a9a430b87f772625366f102 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2dfd0e53a9a430b87f772625366f102" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 39509 } }
I20260811 22:25:53.362365 7970 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2dfd0e53a9a430b87f772625366f102 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:25:53.364524 7984 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2dfd0e53a9a430b87f772625366f102 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2dfd0e53a9a430b87f772625366f102" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2dfd0e53a9a430b87f772625366f102" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 39509 } } }
I20260811 22:25:53.364494 7985 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2dfd0e53a9a430b87f772625366f102 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2dfd0e53a9a430b87f772625366f102. Latest consensus state: current_term: 1 leader_uuid: "d2dfd0e53a9a430b87f772625366f102" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2dfd0e53a9a430b87f772625366f102" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 39509 } } }
I20260811 22:25:53.364908 7984 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2dfd0e53a9a430b87f772625366f102 [sys.catalog]: This master's current role is: LEADER
I20260811 22:25:53.364909 7985 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2dfd0e53a9a430b87f772625366f102 [sys.catalog]: This master's current role is: LEADER
I20260811 22:25:53.367309 7991 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:25:53.372438 7991 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:25:53.380820 7991 catalog_manager.cc:1347] Generated new cluster ID: 422b9d6e80cc411497ee844b762cd09b
I20260811 22:25:53.380908 7991 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:25:53.394658 7991 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:25:53.396332 7991 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:25:53.419812 7991 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2dfd0e53a9a430b87f772625366f102: Generated new TSK 0
I20260811 22:25:53.420697 7991 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:25:53.452498 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:53.443888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47882 (local address 127.29.106.62:39509)
0811 22:25:53.444073 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:53.444079 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:53.444464 (+ 385us) server_negotiation.cc:407] Connection header received
0811 22:25:53.444668 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:53.444673 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:53.444757 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:53.444907 (+ 150us) server_negotiation.cc:226] Negotiated authn=JWT
0811 22:25:53.445313 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:53.446411 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:53.448427 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:53.449099 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:53.449195 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:53.449900 (+ 705us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0811 22:25:53.451961 (+ 2061us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46109/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0811 22:25:53.452226 (+ 265us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47882: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46109/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":56}
W20260811 22:25:53.452605 7980 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47882: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46109/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260811 22:25:53.453433 8033 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:25:53.443755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:39509 (local address 127.0.0.1:47882)
0811 22:25:53.444371 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:53.444385 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:53.444497 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:53.444912 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:53.444915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:53.444924 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0811 22:25:53.445179 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:53.445188 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:53.446586 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:53.446593 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:53.448250 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:53.448267 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:53.449406 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:53.449412 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:53.449584 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:53.449625 (+ 41us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0811 22:25:53.452828 (+ 3203us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46109/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0811 22:25:53.453070 (+ 242us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.29.106.62:39509: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46109/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":348,"threads_started":1}
I20260811 22:25:53.457943 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 7852
I20260811 22:25:53.458987 7965 generic_service.cc:196] Checking for leaks (request via RPC)
E20260811 22:25:53.475936 8036 webserver.cc:533] Webserver: sslize failed:
E20260811 22:25:53.476050 8036 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260811 22:25:53.557495 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 7852
2026-08-11T22:25:53Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (724 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-11T22:25:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:25:53Z Disabled control of system clock
I20260811 22:25:53.644352 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:38615
--webserver_interface=127.29.106.62
--webserver_port=0
--builtin_ntp_servers=127.29.106.20:41191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:38615
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786487105757681-30120-0/testchainca.cert with env {}
W20260811 22:25:53.966589 8043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:53.966861 8043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:53.966954 8043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:53.973999 8043 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:25:53.974102 8043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:53.974151 8043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:53.974213 8043 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:25:53.974257 8043 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260811 22:25:53.974316 8043 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260811 22:25:53.974356 8043 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260811 22:25:53.983879 8043 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:41191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:38615
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:38615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.106.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786487105757681-30120-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:25:53.984997 8043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:53.986192 8043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:53.997460 8050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:53.997432 8057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:53.997394 8051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:53.998106 8043 server_base.cc:1034] running on GCE node
I20260811 22:25:53.998795 8043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:54.000152 8043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:54.001369 8043 hybrid_clock.cc:648] HybridClock initialized: now 1786487154001331 us; error 31 us; skew 500 ppm
I20260811 22:25:54.001729 8043 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 22:25:54.004089 8043 webserver.cc:460] Webserver started at http://127.29.106.62:46273/ using document root <none> and password file <none>
I20260811 22:25:54.004760 8043 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:54.004855 8043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:54.005178 8043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:54.008006 8043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "9dcd503d72f0475f989bf7d61be07e89"
format_stamp: "Formatted at 2026-08-11 22:25:54 on dist-test-slave-gjrz"
server_key: "ed7c8e284f93af9a9547836caf958903"
server_key_iv: "6ae5ed8f6800bd5c36f1cb5ffc42d3f5"
server_key_version: "encryptionkey@0"
I20260811 22:25:54.008795 8043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "9dcd503d72f0475f989bf7d61be07e89"
format_stamp: "Formatted at 2026-08-11 22:25:54 on dist-test-slave-gjrz"
server_key: "ed7c8e284f93af9a9547836caf958903"
server_key_iv: "6ae5ed8f6800bd5c36f1cb5ffc42d3f5"
server_key_version: "encryptionkey@0"
I20260811 22:25:54.013674 8043 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260811 22:25:54.016935 8070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:54.018177 8043 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 22:25:54.018337 8043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "9dcd503d72f0475f989bf7d61be07e89"
format_stamp: "Formatted at 2026-08-11 22:25:54 on dist-test-slave-gjrz"
server_key: "ed7c8e284f93af9a9547836caf958903"
server_key_iv: "6ae5ed8f6800bd5c36f1cb5ffc42d3f5"
server_key_version: "encryptionkey@0"
I20260811 22:25:54.018527 8043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:54.040498 8043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:54.041263 8043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:54.041525 8043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:54.059242 8043 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.62:38615
I20260811 22:25:54.059293 8180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.62:38615 every 8 connection(s)
I20260811 22:25:54.060511 8043 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:25:54.063236 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 8043
I20260811 22:25:54.063454 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:25:54.064083 30120 external_mini_cluster.cc:1428] Setting key c756a40265b985b0bf6da94685bfa329
I20260811 22:25:54.064199 8182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:54.079008 8187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:54.065912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36316 (local address 127.29.106.62:38615)
0811 22:25:54.067001 (+ 1089us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:54.067015 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:54.067080 (+ 65us) server_negotiation.cc:407] Connection header received
0811 22:25:54.069097 (+ 2017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:54.069126 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:54.069703 (+ 577us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:54.070140 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:54.070685 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:54.072670 (+ 1985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:54.073922 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:54.074931 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:54.075140 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:54.076236 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:54.076250 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:54.076268 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:25:54.076979 (+ 711us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:54.077095 (+ 116us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:25:54.077126 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:54.077247 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:54.077487 (+ 240us) server_negotiation.cc:299] Negotiation successful
0811 22:25:54.077790 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":337,"threads_started":1}
I20260811 22:25:54.079355 8182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dcd503d72f0475f989bf7d61be07e89: Bootstrap starting.
I20260811 22:25:54.082499 8182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9dcd503d72f0475f989bf7d61be07e89: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:54.082770 30120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:25:54.083535 8182 log.cc:824] T 00000000000000000000000000000000 P 9dcd503d72f0475f989bf7d61be07e89: Log is configured to *not* fsync() on all Append() calls
I20260811 22:25:54.086390 8182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dcd503d72f0475f989bf7d61be07e89: No bootstrap required, opened a new log
I20260811 22:25:54.094059 8182 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9dcd503d72f0475f989bf7d61be07e89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dcd503d72f0475f989bf7d61be07e89" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 38615 } }
I20260811 22:25:54.094381 8182 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9dcd503d72f0475f989bf7d61be07e89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:25:54.094482 8182 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9dcd503d72f0475f989bf7d61be07e89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dcd503d72f0475f989bf7d61be07e89, State: Initialized, Role: FOLLOWER
I20260811 22:25:54.095011 8182 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9dcd503d72f0475f989bf7d61be07e89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dcd503d72f0475f989bf7d61be07e89" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 38615 } }
I20260811 22:25:54.095221 8182 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9dcd503d72f0475f989bf7d61be07e89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:25:54.095376 8182 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9dcd503d72f0475f989bf7d61be07e89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:25:54.095520 8182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9dcd503d72f0475f989bf7d61be07e89 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:54.097558 8182 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9dcd503d72f0475f989bf7d61be07e89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dcd503d72f0475f989bf7d61be07e89" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 38615 } }
I20260811 22:25:54.098038 8182 leader_election.cc:302] T 00000000000000000000000000000000 P 9dcd503d72f0475f989bf7d61be07e89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9dcd503d72f0475f989bf7d61be07e89; no voters:
I20260811 22:25:54.098610 8182 leader_election.cc:288] T 00000000000000000000000000000000 P 9dcd503d72f0475f989bf7d61be07e89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:25:54.098718 8190 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9dcd503d72f0475f989bf7d61be07e89 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:25:54.099378 8190 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9dcd503d72f0475f989bf7d61be07e89 [term 1 LEADER]: Becoming Leader. State: Replica: 9dcd503d72f0475f989bf7d61be07e89, State: Running, Role: LEADER
I20260811 22:25:54.099951 8190 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9dcd503d72f0475f989bf7d61be07e89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dcd503d72f0475f989bf7d61be07e89" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 38615 } }
I20260811 22:25:54.100688 8182 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9dcd503d72f0475f989bf7d61be07e89 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:25:54.102671 8196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dcd503d72f0475f989bf7d61be07e89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dcd503d72f0475f989bf7d61be07e89. Latest consensus state: current_term: 1 leader_uuid: "9dcd503d72f0475f989bf7d61be07e89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dcd503d72f0475f989bf7d61be07e89" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 38615 } } }
I20260811 22:25:54.103044 8196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dcd503d72f0475f989bf7d61be07e89 [sys.catalog]: This master's current role is: LEADER
I20260811 22:25:54.102864 8193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dcd503d72f0475f989bf7d61be07e89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9dcd503d72f0475f989bf7d61be07e89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dcd503d72f0475f989bf7d61be07e89" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 38615 } } }
I20260811 22:25:54.103143 8193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dcd503d72f0475f989bf7d61be07e89 [sys.catalog]: This master's current role is: LEADER
I20260811 22:25:54.104836 8205 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:25:54.109745 8205 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:25:54.118875 8205 catalog_manager.cc:1347] Generated new cluster ID: e347564379f8498ea6f48263104d40b8
I20260811 22:25:54.118983 8205 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:25:54.146348 8205 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:25:54.147716 8205 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:25:54.156442 8205 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9dcd503d72f0475f989bf7d61be07e89: Generated new TSK 0
I20260811 22:25:54.157188 8205 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:25:54.588644 8187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:54.171642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36320 (local address 127.29.106.62:38615)
0811 22:25:54.171842 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:54.171849 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:54.172337 (+ 488us) server_negotiation.cc:407] Connection header received
0811 22:25:54.172575 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:54.172579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:54.172744 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:54.172880 (+ 136us) server_negotiation.cc:226] Negotiated authn=JWT
0811 22:25:54.173417 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:54.174134 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:54.176219 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:54.176677 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:54.176775 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:54.177333 (+ 558us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0811 22:25:54.588071 (+410738us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0811 22:25:54.588418 (+ 347us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36320: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":61}
W20260811 22:25:54.588785 8187 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36320: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260811 22:25:54.588935 8234 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:25:54.171643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:38615 (local address 127.0.0.1:36320)
0811 22:25:54.172230 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:54.172250 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:54.172394 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:54.172921 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:54.172927 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:54.172940 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0811 22:25:54.173266 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:54.173279 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:54.174273 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:54.174279 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:54.176060 (+ 1781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:54.176076 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:54.176866 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:54.176873 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:54.177052 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:54.177095 (+ 43us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0811 22:25:54.588369 (+411274us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0811 22:25:54.588609 (+ 240us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.29.106.62:38615: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":291,"threads_started":1}
I20260811 22:25:54.590142 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 8043
I20260811 22:25:54.591339 8169 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:25:54.694039 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 8043
2026-08-11T22:25:54Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1120 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (426 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (416 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:25:55 dist-test-slave-gjrz krb5kdc[8274](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:25:55 dist-test-slave-gjrz krb5kdc[8274](info): set up 2 sockets
Aug 11 22:25:55 dist-test-slave-gjrz krb5kdc[8274](info): commencing operation
krb5kdc: starting...
W20260811 22:25:56.789816 30120 mini_kdc.cc:121] Time spent starting KDC: real 1.209s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Aug 11 22:25:56 dist-test-slave-gjrz krb5kdc[8274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:25:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:25:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.106.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
I20260811 22:25:57.012285 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:36451
--webserver_interface=127.29.106.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:38139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:36451
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:25:57.339478 8297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:57.339767 8297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:57.339905 8297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:57.347146 8297 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:25:57.347252 8297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:57.347332 8297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:57.347412 8297 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:25:57.347476 8297 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:25:57.357254 8297 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:38139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:36451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:36451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.106.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:25:57.358536 8297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:57.359853 8297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:57.371116 8309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:57.371150 8307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:57.371115 8306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:57.372269 8297 server_base.cc:1034] running on GCE node
I20260811 22:25:57.372900 8297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:57.374135 8297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:57.375360 8297 hybrid_clock.cc:648] HybridClock initialized: now 1786487157375315 us; error 30 us; skew 500 ppm
I20260811 22:25:57.375701 8297 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:25:57 dist-test-slave-gjrz krb5kdc[8274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487157, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:57.378671 8297 init.cc:377] Logged in from keytab as kudu/127.29.106.62@KRBTEST.COM (short username kudu)
I20260811 22:25:57.380191 8297 webserver.cc:460] Webserver started at http://127.29.106.62:43441/ using document root <none> and password file <none>
I20260811 22:25:57.380839 8297 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:57.380952 8297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:57.381234 8297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:57.384318 8297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "966855c77b6345a489b7fd05a015120d"
format_stamp: "Formatted at 2026-08-11 22:25:57 on dist-test-slave-gjrz"
server_key: "2311bcdb6ae7fec0e7e26761e09d6d85"
server_key_iv: "d6597403db7e5a0e464f68f228137b71"
server_key_version: "encryptionkey@0"
I20260811 22:25:57.385133 8297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "966855c77b6345a489b7fd05a015120d"
format_stamp: "Formatted at 2026-08-11 22:25:57 on dist-test-slave-gjrz"
server_key: "2311bcdb6ae7fec0e7e26761e09d6d85"
server_key_iv: "d6597403db7e5a0e464f68f228137b71"
server_key_version: "encryptionkey@0"
I20260811 22:25:57.390089 8297 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260811 22:25:57.393317 8327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:57.394526 8297 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 22:25:57.394683 8297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "966855c77b6345a489b7fd05a015120d"
format_stamp: "Formatted at 2026-08-11 22:25:57 on dist-test-slave-gjrz"
server_key: "2311bcdb6ae7fec0e7e26761e09d6d85"
server_key_iv: "d6597403db7e5a0e464f68f228137b71"
server_key_version: "encryptionkey@0"
I20260811 22:25:57.394850 8297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:57.413129 8297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:57.416723 8297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:57.416985 8297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:57.434736 8297 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.62:36451
I20260811 22:25:57.434783 8424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.62:36451 every 8 connection(s)
I20260811 22:25:57.436062 8297 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:25:57.439705 8429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:57.442924 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 8297
I20260811 22:25:57.443179 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:25:57.443976 30120 external_mini_cluster.cc:1428] Setting key 093b96f140cdd4eacdc84d4bcab747af
I20260811 22:25:57.452334 8429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 966855c77b6345a489b7fd05a015120d: Bootstrap starting.
I20260811 22:25:57.455873 8429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 966855c77b6345a489b7fd05a015120d: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:57.456845 8429 log.cc:824] T 00000000000000000000000000000000 P 966855c77b6345a489b7fd05a015120d: Log is configured to *not* fsync() on all Append() calls
Aug 11 22:25:57 dist-test-slave-gjrz krb5kdc[8274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:25:57.459585 8429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 966855c77b6345a489b7fd05a015120d: No bootstrap required, opened a new log
I20260811 22:25:57.467808 8437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:57.445954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60772 (local address 127.29.106.62:36451)
0811 22:25:57.446683 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:57.446693 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:57.446734 (+ 41us) server_negotiation.cc:407] Connection header received
0811 22:25:57.448423 (+ 1689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:57.448445 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:57.448818 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:57.449132 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:57.450825 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:57.452611 (+ 1786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:57.454402 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:57.455116 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:57.455292 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:57.459185 (+ 3893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:57.459193 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:57.459207 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:57.459248 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:57.463228 (+ 3980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:57.464155 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:57.464196 (+ 41us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:57.464205 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:57.464472 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:57.464865 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:57.464870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:57.464874 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:57.465511 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:57.465821 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:57.466283 (+ 462us) server_negotiation.cc:299] Negotiation successful
0811 22:25:57.466707 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":336,"threads_started":1}
I20260811 22:25:57.469023 8429 raft_consensus.cc:348] T 00000000000000000000000000000000 P 966855c77b6345a489b7fd05a015120d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "966855c77b6345a489b7fd05a015120d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 36451 } }
I20260811 22:25:57.469381 8429 raft_consensus.cc:374] T 00000000000000000000000000000000 P 966855c77b6345a489b7fd05a015120d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:25:57.469480 8429 raft_consensus.cc:729] T 00000000000000000000000000000000 P 966855c77b6345a489b7fd05a015120d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 966855c77b6345a489b7fd05a015120d, State: Initialized, Role: FOLLOWER
I20260811 22:25:57.470018 8429 consensus_queue.cc:260] T 00000000000000000000000000000000 P 966855c77b6345a489b7fd05a015120d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "966855c77b6345a489b7fd05a015120d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 36451 } }
I20260811 22:25:57.470255 8429 raft_consensus.cc:388] T 00000000000000000000000000000000 P 966855c77b6345a489b7fd05a015120d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:25:57.470407 8429 raft_consensus.cc:482] T 00000000000000000000000000000000 P 966855c77b6345a489b7fd05a015120d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:25:57.470599 8429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 966855c77b6345a489b7fd05a015120d [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:57.472750 8429 raft_consensus.cc:504] T 00000000000000000000000000000000 P 966855c77b6345a489b7fd05a015120d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "966855c77b6345a489b7fd05a015120d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 36451 } }
I20260811 22:25:57.473215 8429 leader_election.cc:302] T 00000000000000000000000000000000 P 966855c77b6345a489b7fd05a015120d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 966855c77b6345a489b7fd05a015120d; no voters:
I20260811 22:25:57.473765 8429 leader_election.cc:288] T 00000000000000000000000000000000 P 966855c77b6345a489b7fd05a015120d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:25:57.473884 8440 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 966855c77b6345a489b7fd05a015120d [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:25:57.474987 8440 raft_consensus.cc:686] T 00000000000000000000000000000000 P 966855c77b6345a489b7fd05a015120d [term 1 LEADER]: Becoming Leader. State: Replica: 966855c77b6345a489b7fd05a015120d, State: Running, Role: LEADER
I20260811 22:25:57.475471 8440 consensus_queue.cc:237] T 00000000000000000000000000000000 P 966855c77b6345a489b7fd05a015120d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "966855c77b6345a489b7fd05a015120d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 36451 } }
I20260811 22:25:57.475878 8429 sys_catalog.cc:564] T 00000000000000000000000000000000 P 966855c77b6345a489b7fd05a015120d [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:25:57.478101 8441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 966855c77b6345a489b7fd05a015120d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "966855c77b6345a489b7fd05a015120d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "966855c77b6345a489b7fd05a015120d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 36451 } } }
I20260811 22:25:57.478192 8443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 966855c77b6345a489b7fd05a015120d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 966855c77b6345a489b7fd05a015120d. Latest consensus state: current_term: 1 leader_uuid: "966855c77b6345a489b7fd05a015120d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "966855c77b6345a489b7fd05a015120d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 36451 } } }
I20260811 22:25:57.478554 8443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 966855c77b6345a489b7fd05a015120d [sys.catalog]: This master's current role is: LEADER
I20260811 22:25:57.478571 8441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 966855c77b6345a489b7fd05a015120d [sys.catalog]: This master's current role is: LEADER
I20260811 22:25:57.483780 8456 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:25:57.488705 8456 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:25:57.498490 8456 catalog_manager.cc:1347] Generated new cluster ID: f89055b92fcf43699a564ae2070fbd10
I20260811 22:25:57.498679 8456 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:25:57.508127 8456 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:25:57.509719 8456 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:25:57.519215 8456 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 966855c77b6345a489b7fd05a015120d: Generated new TSK 0
I20260811 22:25:57.520241 8456 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.106.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab.
Entry for principal kudu/127.29.106.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab.
I20260811 22:25:57.583530 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.1:0
--local_ip_for_outbound_sockets=127.29.106.1
--webserver_interface=127.29.106.1
--webserver_port=0
--tserver_master_addrs=127.29.106.62:36451
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab
--principal=kudu/127.29.106.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:38139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:25:57.910568 8491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:57.910873 8491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:57.911013 8491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:57.918380 8491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:57.918505 8491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:57.918671 8491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.1
I20260811 22:25:57.928458 8491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:38139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab
--principal=kudu/127.29.106.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.106.1
--webserver_port=0
--tserver_master_addrs=127.29.106.62:36451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.1
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:25:57.929674 8491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:57.930935 8491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:57.943794 8498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:57.943738 8502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:57.943783 8497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:57.944670 8491 server_base.cc:1034] running on GCE node
I20260811 22:25:57.945292 8491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:57.946260 8491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:57.947458 8491 hybrid_clock.cc:648] HybridClock initialized: now 1786487157947399 us; error 62 us; skew 500 ppm
I20260811 22:25:57.947762 8491 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:25:57 dist-test-slave-gjrz krb5kdc[8274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487157, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:57.950726 8491 init.cc:377] Logged in from keytab as kudu/127.29.106.1@KRBTEST.COM (short username kudu)
I20260811 22:25:57.952159 8491 webserver.cc:460] Webserver started at http://127.29.106.1:45011/ using document root <none> and password file <none>
I20260811 22:25:57.952777 8491 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:57.952875 8491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:57.953122 8491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:57.955987 8491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/ts-0/data/instance:
uuid: "59944b28852242c1bc3ab29389b07d30"
format_stamp: "Formatted at 2026-08-11 22:25:57 on dist-test-slave-gjrz"
server_key: "a8904654ca8d020b2ec648f2cee1a069"
server_key_iv: "0ea501a057c401a90c99a89b2e434d6f"
server_key_version: "encryptionkey@0"
I20260811 22:25:57.956732 8491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance:
uuid: "59944b28852242c1bc3ab29389b07d30"
format_stamp: "Formatted at 2026-08-11 22:25:57 on dist-test-slave-gjrz"
server_key: "a8904654ca8d020b2ec648f2cee1a069"
server_key_iv: "0ea501a057c401a90c99a89b2e434d6f"
server_key_version: "encryptionkey@0"
I20260811 22:25:57.961488 8491 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 22:25:57.964685 8520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:57.965927 8491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260811 22:25:57.966073 8491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/ts-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/ts-0/wal
uuid: "59944b28852242c1bc3ab29389b07d30"
format_stamp: "Formatted at 2026-08-11 22:25:57 on dist-test-slave-gjrz"
server_key: "a8904654ca8d020b2ec648f2cee1a069"
server_key_iv: "0ea501a057c401a90c99a89b2e434d6f"
server_key_version: "encryptionkey@0"
I20260811 22:25:57.966284 8491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:57.987685 8491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:57.991473 8491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:57.991747 8491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:57.992678 8491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:25:57.993944 8491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:25:57.994016 8491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:57.994141 8491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:25:57.994194 8491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:58.025275 8491 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.1:36467
I20260811 22:25:58.025328 8755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.1:36467 every 8 connection(s)
I20260811 22:25:58.026515 8491 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
I20260811 22:25:58.027200 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 8491
I20260811 22:25:58.027464 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance
I20260811 22:25:58.028239 30120 external_mini_cluster.cc:1428] Setting key 82ba6c7ee0a7282104ec62d8e4cb8a43
Aug 11 22:25:58 dist-test-slave-gjrz krb5kdc[8274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487157, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.1@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:25:58.046306 8758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:58.029420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:43329 (local address 127.29.106.62:36451)
0811 22:25:58.029878 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:58.029884 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:58.030848 (+ 964us) server_negotiation.cc:407] Connection header received
0811 22:25:58.032696 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:58.032701 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:58.032776 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:58.032867 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:58.034321 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:58.035029 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:58.036576 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:58.036965 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:58.037082 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:58.040787 (+ 3705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:58.040792 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:58.040795 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:58.040826 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:58.043171 (+ 2345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:58.044048 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:58.044053 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:58.044056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:58.044208 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:58.044702 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:58.044705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:58.044707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:58.044964 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:58.045078 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:58.045882 (+ 804us) server_negotiation.cc:299] Negotiation successful
0811 22:25:58.046052 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":223,"threads_started":1}
I20260811 22:25:58.047619 8759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:58.029892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:36451 (local address 127.29.106.1:43329)
0811 22:25:58.030698 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:58.030736 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:58.032429 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:58.033116 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:58.033124 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:58.033618 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:58.034151 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:58.034169 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:58.035186 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:58.035190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:58.036430 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:58.036443 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:58.037190 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:58.037197 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:58.037412 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:58.038020 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:58.038048 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:58.040579 (+ 2531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:58.043334 (+ 2755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:58.043342 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:58.043357 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:58.043883 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:58.044336 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:58.044342 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:58.044345 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:58.044604 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:58.045096 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:58.045103 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:58.045479 (+ 376us) client_negotiation.cc:769] Sending connection context
0811 22:25:58.045786 (+ 307us) client_negotiation.cc:240] Negotiation successful
0811 22:25:58.046294 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":351,"threads_started":1}
I20260811 22:25:58.049067 8756 heartbeater.cc:347] Connected to a master server at 127.29.106.62:36451
I20260811 22:25:58.049331 8756 heartbeater.cc:464] Registering TS with master...
I20260811 22:25:58.049980 8756 heartbeater.cc:511] Master 127.29.106.62:36451 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.29.106.2@KRBTEST.COM; defaulting to no policy
I20260811 22:25:58.051988 8358 ts_manager.cc:194] Registered new tserver with Master: 59944b28852242c1bc3ab29389b07d30 (127.29.106.1:36467)
I20260811 22:25:58.053385 8358 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.1@KRBTEST.COM'} at 127.29.106.1:43329
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab.
Entry for principal kudu/127.29.106.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab.
I20260811 22:25:58.080580 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.2:0
--local_ip_for_outbound_sockets=127.29.106.2
--webserver_interface=127.29.106.2
--webserver_port=0
--tserver_master_addrs=127.29.106.62:36451
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab
--principal=kudu/127.29.106.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:38139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:25:58.398770 8765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:58.399039 8765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:58.399170 8765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:58.406523 8765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:58.406654 8765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:58.406816 8765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.2
I20260811 22:25:58.416412 8765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:38139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab
--principal=kudu/127.29.106.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.106.2
--webserver_port=0
--tserver_master_addrs=127.29.106.62:36451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.2
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:25:58.417579 8765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:58.419248 8765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:58.432137 8775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:58.432173 8779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:58.432224 8777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:58.433033 8765 server_base.cc:1034] running on GCE node
I20260811 22:25:58.433574 8765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:58.434440 8765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:58.435623 8765 hybrid_clock.cc:648] HybridClock initialized: now 1786487158435573 us; error 43 us; skew 500 ppm
I20260811 22:25:58.435881 8765 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:25:58 dist-test-slave-gjrz krb5kdc[8274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487158, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:58.438863 8765 init.cc:377] Logged in from keytab as kudu/127.29.106.2@KRBTEST.COM (short username kudu)
I20260811 22:25:58.440245 8765 webserver.cc:460] Webserver started at http://127.29.106.2:35901/ using document root <none> and password file <none>
I20260811 22:25:58.440871 8765 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:58.440968 8765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:58.441255 8765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:58.444027 8765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/ts-1/data/instance:
uuid: "f0314fa240454e708f97d15ef7854b0a"
format_stamp: "Formatted at 2026-08-11 22:25:58 on dist-test-slave-gjrz"
server_key: "dc154d6692862d8320cef4605d2eba7a"
server_key_iv: "c3f55964ebcac9da57a98982dacb0778"
server_key_version: "encryptionkey@0"
I20260811 22:25:58.444756 8765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance:
uuid: "f0314fa240454e708f97d15ef7854b0a"
format_stamp: "Formatted at 2026-08-11 22:25:58 on dist-test-slave-gjrz"
server_key: "dc154d6692862d8320cef4605d2eba7a"
server_key_iv: "c3f55964ebcac9da57a98982dacb0778"
server_key_version: "encryptionkey@0"
I20260811 22:25:58.449410 8765 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 22:25:58.452565 8792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:58.453711 8765 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:25:58.453850 8765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/ts-1/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/ts-1/wal
uuid: "f0314fa240454e708f97d15ef7854b0a"
format_stamp: "Formatted at 2026-08-11 22:25:58 on dist-test-slave-gjrz"
server_key: "dc154d6692862d8320cef4605d2eba7a"
server_key_iv: "c3f55964ebcac9da57a98982dacb0778"
server_key_version: "encryptionkey@0"
I20260811 22:25:58.454015 8765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:58.470955 8765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:58.475090 8765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:58.475401 8765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:58.476410 8765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:25:58.477711 8765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:25:58.477784 8765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:58.477875 8765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:25:58.477917 8765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:58.507803 8765 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.2:34443
I20260811 22:25:58.507874 9034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.2:34443 every 8 connection(s)
I20260811 22:25:58.509039 8765 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
I20260811 22:25:58.509876 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 8765
I20260811 22:25:58.510078 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance
I20260811 22:25:58.510629 30120 external_mini_cluster.cc:1428] Setting key f63f674cb8ac07a90ae4de4a77049050
Aug 11 22:25:58 dist-test-slave-gjrz krb5kdc[8274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487158, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.2@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:25:58.530182 8758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:58.511889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:38505 (local address 127.29.106.62:36451)
0811 22:25:58.512084 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:58.512090 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:58.513330 (+ 1240us) server_negotiation.cc:407] Connection header received
0811 22:25:58.515764 (+ 2434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:58.515770 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:58.515850 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:58.515953 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:58.517746 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:58.518428 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:58.520553 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:58.520965 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:58.521058 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:58.524696 (+ 3638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:58.524701 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:58.524704 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:58.524734 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:58.527117 (+ 2383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:58.528007 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:58.528011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:58.528013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:58.528143 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:58.528587 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:58.528591 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:58.528593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:58.528849 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:58.528977 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:58.529641 (+ 664us) server_negotiation.cc:299] Negotiation successful
0811 22:25:58.529862 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260811 22:25:58.530995 9040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:58.512346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:36451 (local address 127.29.106.2:38505)
0811 22:25:58.513179 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:58.513225 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:58.515437 (+ 2212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:58.516210 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:58.516220 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:58.516856 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:58.517591 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:58.517617 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:58.518590 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:58.518595 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:58.520402 (+ 1807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:58.520418 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:58.521183 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:58.521189 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:58.521452 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:58.522101 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:58.522128 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:58.524484 (+ 2356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:58.527253 (+ 2769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:58.527288 (+ 35us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:58.527302 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:58.527897 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:58.528267 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:58.528272 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:58.528276 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:58.528459 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:58.528977 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:58.528985 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:58.529285 (+ 300us) client_negotiation.cc:769] Sending connection context
0811 22:25:58.529509 (+ 224us) client_negotiation.cc:240] Negotiation successful
0811 22:25:58.529967 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":2868,"threads_started":1}
I20260811 22:25:58.532306 9035 heartbeater.cc:347] Connected to a master server at 127.29.106.62:36451
I20260811 22:25:58.532629 9035 heartbeater.cc:464] Registering TS with master...
I20260811 22:25:58.533480 9035 heartbeater.cc:511] Master 127.29.106.62:36451 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.29.106.3@KRBTEST.COM; defaulting to no policy
I20260811 22:25:58.535086 8358 ts_manager.cc:194] Registered new tserver with Master: f0314fa240454e708f97d15ef7854b0a (127.29.106.2:34443)
I20260811 22:25:58.535923 8358 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.2@KRBTEST.COM'} at 127.29.106.2:38505
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab.
Entry for principal kudu/127.29.106.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab.
I20260811 22:25:58.566195 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.3:0
--local_ip_for_outbound_sockets=127.29.106.3
--webserver_interface=127.29.106.3
--webserver_port=0
--tserver_master_addrs=127.29.106.62:36451
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab
--principal=kudu/127.29.106.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:38139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:25:58.928020 9044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:58.928283 9044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:58.928418 9044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:58.935457 9044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:58.935557 9044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:58.935706 9044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.3
I20260811 22:25:58.945348 9044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:38139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab
--principal=kudu/127.29.106.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.106.3
--webserver_port=0
--tserver_master_addrs=127.29.106.62:36451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.3
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:25:58.946697 9044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:58.948093 9044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:58.961043 9054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:58.961092 9055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:58.961493 9058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:58.962309 9044 server_base.cc:1034] running on GCE node
I20260811 22:25:58.962913 9044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:58.963886 9044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:58.965091 9044 hybrid_clock.cc:648] HybridClock initialized: now 1786487158965036 us; error 38 us; skew 500 ppm
I20260811 22:25:58.965385 9044 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:25:58 dist-test-slave-gjrz krb5kdc[8274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487158, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:58.968580 9044 init.cc:377] Logged in from keytab as kudu/127.29.106.3@KRBTEST.COM (short username kudu)
I20260811 22:25:58.970237 9044 webserver.cc:460] Webserver started at http://127.29.106.3:41381/ using document root <none> and password file <none>
I20260811 22:25:58.970907 9044 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:58.971042 9044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:58.971413 9044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:58.974306 9044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/ts-2/data/instance:
uuid: "9a3c4111d9734a9b9367370d484b6fff"
format_stamp: "Formatted at 2026-08-11 22:25:58 on dist-test-slave-gjrz"
server_key: "2cb71cb35d7610d1b6473e58e9159692"
server_key_iv: "481d5555d1be0d5106248229627c604d"
server_key_version: "encryptionkey@0"
I20260811 22:25:58.975097 9044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance:
uuid: "9a3c4111d9734a9b9367370d484b6fff"
format_stamp: "Formatted at 2026-08-11 22:25:58 on dist-test-slave-gjrz"
server_key: "2cb71cb35d7610d1b6473e58e9159692"
server_key_iv: "481d5555d1be0d5106248229627c604d"
server_key_version: "encryptionkey@0"
I20260811 22:25:58.980450 9044 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260811 22:25:58.984349 9075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:58.985770 9044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260811 22:25:58.985962 9044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/ts-2/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/ts-2/wal
uuid: "9a3c4111d9734a9b9367370d484b6fff"
format_stamp: "Formatted at 2026-08-11 22:25:58 on dist-test-slave-gjrz"
server_key: "2cb71cb35d7610d1b6473e58e9159692"
server_key_iv: "481d5555d1be0d5106248229627c604d"
server_key_version: "encryptionkey@0"
I20260811 22:25:58.986168 9044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:59.003333 9044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:59.007297 9044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:59.007576 9044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:59.008503 9044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:25:59.009822 9044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:25:59.009900 9044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:59.010054 9044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:25:59.010130 9044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:59.041924 9044 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.3:40117
I20260811 22:25:59.041982 9321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.3:40117 every 8 connection(s)
I20260811 22:25:59.043133 9044 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
I20260811 22:25:59.048727 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 9044
I20260811 22:25:59.048902 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance
I20260811 22:25:59.049443 30120 external_mini_cluster.cc:1428] Setting key 069d3699775c3afb9c6d1472c33fbcb8
I20260811 22:25:59.056627 8756 heartbeater.cc:503] Master 127.29.106.62:36451 was elected leader, sending a full tablet report...
Aug 11 22:25:59 dist-test-slave-gjrz krb5kdc[8274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487158, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.3@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:25:59.063364 9330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:59.045778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:44903 (local address 127.29.106.62:36451)
0811 22:25:59.046246 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:59.046254 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:59.047251 (+ 997us) server_negotiation.cc:407] Connection header received
0811 22:25:59.049187 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:59.049193 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:59.049279 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:59.049397 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:59.051195 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:59.052097 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:59.053892 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:59.054311 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:59.054407 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:59.057725 (+ 3318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:59.057731 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:59.057735 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:59.057768 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:59.060404 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:59.061080 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:59.061084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:59.061086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:59.061213 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:59.061678 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:59.061682 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:59.061684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:59.062008 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:59.062140 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:59.062831 (+ 691us) server_negotiation.cc:299] Negotiation successful
0811 22:25:59.063026 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":217,"threads_started":1}
I20260811 22:25:59.064620 9332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:59.046275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:36451 (local address 127.29.106.3:44903)
0811 22:25:59.047090 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:59.047142 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:59.048901 (+ 1759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:59.049654 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:59.049662 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:59.050283 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:59.051005 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:59.051027 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:59.052282 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:59.052288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:59.053754 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:59.053767 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:59.054506 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:59.054511 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:59.054699 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:59.055323 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:59.055364 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:59.057529 (+ 2165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:59.060546 (+ 3017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:59.060554 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:59.060567 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:59.060981 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:59.061338 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:59.061342 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:59.061344 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:59.061541 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:59.062152 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:59.062160 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:59.062513 (+ 353us) client_negotiation.cc:769] Sending connection context
0811 22:25:59.062757 (+ 244us) client_negotiation.cc:240] Negotiation successful
0811 22:25:59.063340 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":334,"threads_started":1}
I20260811 22:25:59.066037 9325 heartbeater.cc:347] Connected to a master server at 127.29.106.62:36451
I20260811 22:25:59.066331 9325 heartbeater.cc:464] Registering TS with master...
I20260811 22:25:59.067045 9325 heartbeater.cc:511] Master 127.29.106.62:36451 requested a full tablet report, sending...
I20260811 22:25:59.068585 8358 ts_manager.cc:194] Registered new tserver with Master: 9a3c4111d9734a9b9367370d484b6fff (127.29.106.3:40117)
I20260811 22:25:59.069480 8358 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.3@KRBTEST.COM'} at 127.29.106.3:44903
I20260811 22:25:59.077548 30120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/krb5kdc/test-admin.keytab.
Aug 11 22:25:59 dist-test-slave-gjrz krb5kdc[8274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:59.109664 30120 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260811 22:25:59.114773 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:25:59.112334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46386 (local address 127.29.106.1:36467)
0811 22:25:59.112976 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:59.112985 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:59.113018 (+ 33us) server_negotiation.cc:407] Connection header received
0811 22:25:59.113090 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:59.113097 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:59.113311 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:59.113454 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:59.114368 (+ 914us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46386: BlockingRecv error: recv got EOF from 127.0.0.1:46386 (error 108)
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":290,"threads_started":1}
W20260811 22:25:59.114845 30120 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/krb5kdc/krb5cc' of type FILE
Aug 11 22:25:59 dist-test-slave-gjrz krb5kdc[8274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:59.116348 30120 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 11 22:25:59 dist-test-slave-gjrz krb5kdc[8274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:59.117597 30120 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 11 22:25:59 dist-test-slave-gjrz krb5kdc[8274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.1@KRBTEST.COM
I20260811 22:25:59.131417 9349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:59.119127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46388 (local address 127.29.106.1:36467)
0811 22:25:59.119326 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:59.119332 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:59.119470 (+ 138us) server_negotiation.cc:407] Connection header received
0811 22:25:59.119598 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:59.119601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:59.119654 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:59.119726 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:59.120594 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:59.121544 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:59.122662 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:59.123185 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:59.123333 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:59.126489 (+ 3156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:59.126500 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:59.126514 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:59.126545 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:59.129119 (+ 2574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:59.129697 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:59.129703 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:59.129705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:59.129817 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:59.130106 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:59.130109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:59.130111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:59.130444 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:59.130649 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:59.130899 (+ 250us) server_negotiation.cc:299] Negotiation successful
0811 22:25:59.131096 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260811 22:25:59.136794 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:25:59.135128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46402 (local address 127.29.106.1:36467)
0811 22:25:59.135340 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:59.135346 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:59.135518 (+ 172us) server_negotiation.cc:407] Connection header received
0811 22:25:59.135678 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:59.135683 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:59.135736 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:59.135811 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:59.136581 (+ 770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46402: BlockingRecv error: recv got EOF from 127.0.0.1:46402 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260811 22:25:59.136953 30120 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/krb5kdc/krb5cc' of type FILE
Aug 11 22:25:59 dist-test-slave-gjrz krb5kdc[8274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:59.138338 30120 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 11 22:25:59 dist-test-slave-gjrz krb5kdc[8274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:59.139575 30120 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 11 22:25:59 dist-test-slave-gjrz krb5kdc[8274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.1@KRBTEST.COM
I20260811 22:25:59.153120 9349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:59.140767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46404 (local address 127.29.106.1:36467)
0811 22:25:59.140926 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:59.140932 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:59.141220 (+ 288us) server_negotiation.cc:407] Connection header received
0811 22:25:59.141358 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:59.141362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:59.141414 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:59.141491 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:59.142421 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:59.143113 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:59.144347 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:59.144816 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:59.144940 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:59.148469 (+ 3529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:59.148475 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:59.148479 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:59.148513 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:59.150954 (+ 2441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:59.151536 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:59.151539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:59.151542 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:59.151682 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:59.151969 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:59.151972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:59.151974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:59.152225 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:59.152425 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:59.152659 (+ 234us) server_negotiation.cc:299] Negotiation successful
0811 22:25:59.152839 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260811 22:25:59.157367 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:25:59.155661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46418 (local address 127.29.106.1:36467)
0811 22:25:59.155811 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:59.155816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:59.156165 (+ 349us) server_negotiation.cc:407] Connection header received
0811 22:25:59.156295 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:59.156301 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:59.156370 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:59.156444 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:59.157172 (+ 728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46418: BlockingRecv error: recv got EOF from 127.0.0.1:46418 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260811 22:25:59.157893 30120 init.cc:244] Error while opening credential cache '/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786487105757681-30120-0/krb5kdc/krb5cc)
Aug 11 22:25:59 dist-test-slave-gjrz krb5kdc[8274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:59.159371 30120 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 11 22:25:59 dist-test-slave-gjrz krb5kdc[8274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:59.160568 30120 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 11 22:25:59 dist-test-slave-gjrz krb5kdc[8274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.1@KRBTEST.COM
I20260811 22:25:59.173678 9349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:59.161736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46430 (local address 127.29.106.1:36467)
0811 22:25:59.161928 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:59.161933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:59.162158 (+ 225us) server_negotiation.cc:407] Connection header received
0811 22:25:59.162261 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:59.162264 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:59.162314 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:59.162384 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:59.163231 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:59.163950 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:59.165077 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:59.165471 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:59.165554 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:59.168782 (+ 3228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:59.168787 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:59.168790 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:59.168819 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:59.171328 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:59.171905 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:59.171909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:59.171916 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:59.172051 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:59.172464 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:59.172468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:59.172470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:59.172754 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:59.173015 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:59.173164 (+ 149us) server_negotiation.cc:299] Negotiation successful
0811 22:25:59.173388 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260811 22:25:59.175026 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 8491
I20260811 22:25:59.186983 9349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:59.175748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46446 (local address 127.29.106.1:36467)
0811 22:25:59.175933 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:59.175939 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:59.176256 (+ 317us) server_negotiation.cc:407] Connection header received
0811 22:25:59.176396 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:59.176402 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:59.176459 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:59.176532 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:59.177435 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:59.178090 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:59.179254 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:59.179725 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:59.179834 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:59.181833 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:59.181839 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:59.181842 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:59.181888 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:59.184488 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:59.185119 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:59.185124 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:59.185126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:59.185263 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:59.185635 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:59.185645 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:59.185647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:59.185905 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:59.186167 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:59.186467 (+ 300us) server_negotiation.cc:299] Negotiation successful
0811 22:25:59.186680 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260811 22:25:59.187666 8744 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:25:59.270624 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 8491
I20260811 22:25:59.289963 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 8765
Aug 11 22:25:59 dist-test-slave-gjrz krb5kdc[8274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.2@KRBTEST.COM
I20260811 22:25:59.304643 9374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:59.290873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55640 (local address 127.29.106.2:34443)
0811 22:25:59.291433 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:59.291440 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:59.291466 (+ 26us) server_negotiation.cc:407] Connection header received
0811 22:25:59.291528 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:59.291535 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:59.291750 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:59.291876 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:59.292808 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:59.293833 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:59.294968 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:59.295571 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:59.295662 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:59.299110 (+ 3448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:59.299122 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:59.299137 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:59.299168 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:59.302017 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:59.302627 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:59.302634 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:59.302637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:59.302762 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:59.303102 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:59.303105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:59.303107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:59.303556 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:59.303783 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:59.304077 (+ 294us) server_negotiation.cc:299] Negotiation successful
0811 22:25:59.304310 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":280,"threads_started":1}
I20260811 22:25:59.305912 9026 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:25:59.387591 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 8765
I20260811 22:25:59.406204 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 9044
Aug 11 22:25:59 dist-test-slave-gjrz krb5kdc[8274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.3@KRBTEST.COM
I20260811 22:25:59.421350 9380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:59.407137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60186 (local address 127.29.106.3:40117)
0811 22:25:59.407739 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:59.407746 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:59.407777 (+ 31us) server_negotiation.cc:407] Connection header received
0811 22:25:59.407841 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:59.407848 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:59.408064 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:59.408188 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:59.409185 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:59.410264 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:59.411566 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:59.412141 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:59.412237 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:59.415929 (+ 3692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:59.415941 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:59.415955 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:59.415988 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:59.418683 (+ 2695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:59.419456 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:59.419464 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:59.419466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:59.419601 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:59.419952 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:59.419955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:59.419956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:59.420313 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:59.420552 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:59.420774 (+ 222us) server_negotiation.cc:299] Negotiation successful
0811 22:25:59.421030 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":282,"threads_started":1}
I20260811 22:25:59.422662 9308 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:25:59.509198 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 9044
I20260811 22:25:59.531695 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 8297
I20260811 22:25:59.532891 8417 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:25:59.636847 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 8297
2026-08-11T22:25:59Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4103 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:25:59 dist-test-slave-gjrz krb5kdc[9387](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:25:59 dist-test-slave-gjrz krb5kdc[9387](info): set up 2 sockets
Aug 11 22:25:59 dist-test-slave-gjrz krb5kdc[9387](info): commencing operation
krb5kdc: starting...
W20260811 22:26:00.895427 30120 mini_kdc.cc:121] Time spent starting KDC: real 1.212s user 0.006s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Aug 11 22:26:01 dist-test-slave-gjrz krb5kdc[9387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:26:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:26:01Z Disabled control of system clock
WARNING: no policy specified for oryx/127.29.106.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.106.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.106.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.62.keytab.
Entry for principal oryx/127.29.106.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.62.keytab.
I20260811 22:26:01.130601 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:46573
--webserver_interface=127.29.106.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.62.keytab
--principal=oryx/127.29.106.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:34763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:46573
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:26:01.457955 9418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:26:01.458259 9418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:26:01.458396 9418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:26:01.466077 9418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:26:01.466219 9418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:26:01.466306 9418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:26:01.466387 9418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:26:01.466452 9418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:26:01.477552 9418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:34763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:46573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.62.keytab
--principal=oryx/127.29.106.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:46573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.106.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:26:01.479364 9418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:26:01.481002 9418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:26:01.492733 9428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:01.492787 9429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:26:01.493580 9418 server_base.cc:1034] running on GCE node
W20260811 22:26:01.495574 9431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:26:01.496502 9418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:26:01.498147 9418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:26:01.499354 9418 hybrid_clock.cc:648] HybridClock initialized: now 1786487161499283 us; error 51 us; skew 500 ppm
I20260811 22:26:01.499745 9418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:26:01 dist-test-slave-gjrz krb5kdc[9387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487161, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.106.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:26:01.506482 9418 init.cc:377] Logged in from keytab as oryx/127.29.106.62@KRBTEST.COM (short username oryx)
I20260811 22:26:01.508075 9418 webserver.cc:460] Webserver started at http://127.29.106.62:46691/ using document root <none> and password file <none>
I20260811 22:26:01.508762 9418 fs_manager.cc:359] Metadata directory not provided
I20260811 22:26:01.508868 9418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:26:01.509119 9418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:26:01.512221 9418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "2b5d1ed0b161417688f43a878b41ef3d"
format_stamp: "Formatted at 2026-08-11 22:26:01 on dist-test-slave-gjrz"
server_key: "b28b357a80d9a238456bae8f1d677c12"
server_key_iv: "2cbca048cda63263e5c5df7296a21ea7"
server_key_version: "encryptionkey@0"
I20260811 22:26:01.512964 9418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "2b5d1ed0b161417688f43a878b41ef3d"
format_stamp: "Formatted at 2026-08-11 22:26:01 on dist-test-slave-gjrz"
server_key: "b28b357a80d9a238456bae8f1d677c12"
server_key_iv: "2cbca048cda63263e5c5df7296a21ea7"
server_key_version: "encryptionkey@0"
I20260811 22:26:01.518137 9418 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260811 22:26:01.521438 9452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:01.522696 9418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 22:26:01.522857 9418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "2b5d1ed0b161417688f43a878b41ef3d"
format_stamp: "Formatted at 2026-08-11 22:26:01 on dist-test-slave-gjrz"
server_key: "b28b357a80d9a238456bae8f1d677c12"
server_key_iv: "2cbca048cda63263e5c5df7296a21ea7"
server_key_version: "encryptionkey@0"
I20260811 22:26:01.523034 9418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:26:01.544272 9418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:26:01.548586 9418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:26:01.548854 9418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:26:01.566066 9418 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.62:46573
I20260811 22:26:01.566099 9552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.62:46573 every 8 connection(s)
I20260811 22:26:01.567363 9418 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:26:01.571111 9556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:26:01.573036 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 9418
I20260811 22:26:01.573282 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:26:01.574074 30120 external_mini_cluster.cc:1428] Setting key 98a11f50aaf388126f4184a5374d5638
I20260811 22:26:01.583333 9556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d: Bootstrap starting.
I20260811 22:26:01.587230 9556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d: Neither blocks nor log segments found. Creating new log.
I20260811 22:26:01.588554 9556 log.cc:824] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d: Log is configured to *not* fsync() on all Append() calls
Aug 11 22:26:01 dist-test-slave-gjrz krb5kdc[9387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM
I20260811 22:26:01.591511 9556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d: No bootstrap required, opened a new log
I20260811 22:26:01.596645 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:01.575986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42312 (local address 127.29.106.62:46573)
0811 22:26:01.576787 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:01.576798 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:01.576839 (+ 41us) server_negotiation.cc:407] Connection header received
0811 22:26:01.578449 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:01.578470 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:01.578829 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:01.579141 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:01.580994 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:01.582350 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:01.584027 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:01.584721 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:01.584875 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:01.589639 (+ 4764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:01.589648 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:01.589667 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:01.589708 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:01.592631 (+ 2923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:01.593291 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:01.593299 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:01.593306 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:01.593478 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:01.593866 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:01.593870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:01.593872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:01.594411 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:01.594736 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:01.595183 (+ 447us) server_negotiation.cc:299] Negotiation successful
0811 22:26:01.595648 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":380,"threads_started":1}
I20260811 22:26:01.600003 9556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } }
I20260811 22:26:01.600342 9556 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:26:01.600448 9556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b5d1ed0b161417688f43a878b41ef3d, State: Initialized, Role: FOLLOWER
I20260811 22:26:01.601014 9556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } }
I20260811 22:26:01.601253 9556 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:26:01.601380 9556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:26:01.601540 9556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:26:01.604076 9556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } }
I20260811 22:26:01.604601 9556 leader_election.cc:302] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b5d1ed0b161417688f43a878b41ef3d; no voters:
I20260811 22:26:01.605120 9556 leader_election.cc:288] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:26:01.605248 9564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:26:01.605880 9564 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d [term 1 LEADER]: Becoming Leader. State: Replica: 2b5d1ed0b161417688f43a878b41ef3d, State: Running, Role: LEADER
I20260811 22:26:01.606415 9564 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } }
I20260811 22:26:01.606992 9556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:26:01.609315 9565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b5d1ed0b161417688f43a878b41ef3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } }
I20260811 22:26:01.609714 9565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d [sys.catalog]: This master's current role is: LEADER
I20260811 22:26:01.610144 9566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b5d1ed0b161417688f43a878b41ef3d. Latest consensus state: current_term: 1 leader_uuid: "2b5d1ed0b161417688f43a878b41ef3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } }
I20260811 22:26:01.610515 9566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d [sys.catalog]: This master's current role is: LEADER
I20260811 22:26:01.612758 9578 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:26:01.618073 9578 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:26:01.626191 9578 catalog_manager.cc:1347] Generated new cluster ID: 8731d947535c4e8c8f3224f1af27f098
I20260811 22:26:01.626312 9578 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:26:01.637233 9578 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:26:01.639140 9578 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:26:01.653463 9578 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d: Generated new TSK 0
I20260811 22:26:01.654251 9578 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.29.106.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.106.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.106.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.1.keytab.
Entry for principal oryx/127.29.106.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.1.keytab.
I20260811 22:26:01.726624 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.1:0
--local_ip_for_outbound_sockets=127.29.106.1
--webserver_interface=127.29.106.1
--webserver_port=0
--tserver_master_addrs=127.29.106.62:46573
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.1.keytab
--principal=oryx/127.29.106.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:34763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:26:02.064723 9599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:26:02.065022 9599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:26:02.065127 9599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:26:02.072340 9599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:26:02.072453 9599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:26:02.072571 9599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.1
I20260811 22:26:02.082217 9599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:34763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.1.keytab
--principal=oryx/127.29.106.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.106.1
--webserver_port=0
--tserver_master_addrs=127.29.106.62:46573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.1
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:26:02.083448 9599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:26:02.084697 9599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:26:02.097697 9610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:02.097766 9614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:02.097710 9612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:26:02.098631 9599 server_base.cc:1034] running on GCE node
I20260811 22:26:02.099223 9599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:26:02.100211 9599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:26:02.101420 9599 hybrid_clock.cc:648] HybridClock initialized: now 1786487162101380 us; error 26 us; skew 500 ppm
I20260811 22:26:02.101732 9599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:26:02 dist-test-slave-gjrz krb5kdc[9387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487162, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.106.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:26:02.105032 9599 init.cc:377] Logged in from keytab as oryx/127.29.106.1@KRBTEST.COM (short username oryx)
I20260811 22:26:02.106623 9599 webserver.cc:460] Webserver started at http://127.29.106.1:38993/ using document root <none> and password file <none>
I20260811 22:26:02.107422 9599 fs_manager.cc:359] Metadata directory not provided
I20260811 22:26:02.107558 9599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:26:02.107939 9599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:26:02.111614 9599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-0/data/instance:
uuid: "f16cafe5e19446e782da1ed0fcd4a964"
format_stamp: "Formatted at 2026-08-11 22:26:02 on dist-test-slave-gjrz"
server_key: "59c3ff62a5f6d611f812249a934eea37"
server_key_iv: "d735da4e44ebe03d0937e108dfe182ce"
server_key_version: "encryptionkey@0"
I20260811 22:26:02.112528 9599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance:
uuid: "f16cafe5e19446e782da1ed0fcd4a964"
format_stamp: "Formatted at 2026-08-11 22:26:02 on dist-test-slave-gjrz"
server_key: "59c3ff62a5f6d611f812249a934eea37"
server_key_iv: "d735da4e44ebe03d0937e108dfe182ce"
server_key_version: "encryptionkey@0"
I20260811 22:26:02.117787 9599 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260811 22:26:02.121132 9630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:02.122400 9599 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 22:26:02.122550 9599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-0/wal
uuid: "f16cafe5e19446e782da1ed0fcd4a964"
format_stamp: "Formatted at 2026-08-11 22:26:02 on dist-test-slave-gjrz"
server_key: "59c3ff62a5f6d611f812249a934eea37"
server_key_iv: "d735da4e44ebe03d0937e108dfe182ce"
server_key_version: "encryptionkey@0"
I20260811 22:26:02.122721 9599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:26:02.147204 9599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:26:02.151499 9599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:26:02.151799 9599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:26:02.152907 9599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:26:02.154487 9599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:26:02.154570 9599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:02.154667 9599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:26:02.154718 9599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:02.185729 9599 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.1:35211
I20260811 22:26:02.185788 9849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.1:35211 every 8 connection(s)
I20260811 22:26:02.187150 9599 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
I20260811 22:26:02.192603 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 9599
I20260811 22:26:02.192804 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance
I20260811 22:26:02.193393 30120 external_mini_cluster.cc:1428] Setting key 73e9d5488fdcfc3bd2380eb0b964c01d
Aug 11 22:26:02 dist-test-slave-gjrz krb5kdc[9387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487162, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.106.1@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM
I20260811 22:26:02.206584 9853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:02.189835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:60689 (local address 127.29.106.62:46573)
0811 22:26:02.190281 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:02.190286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:02.191242 (+ 956us) server_negotiation.cc:407] Connection header received
0811 22:26:02.193110 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:02.193116 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:02.193207 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:02.193306 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:02.194929 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:02.195813 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:02.197340 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:02.197747 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:02.197862 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:02.201217 (+ 3355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:02.201222 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:02.201225 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:02.201258 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:02.203699 (+ 2441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:02.204381 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:02.204385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:02.204387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:02.204509 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:02.204916 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:02.204919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:02.204921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:02.205176 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:02.205315 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:02.205987 (+ 672us) server_negotiation.cc:299] Negotiation successful
0811 22:26:02.206205 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":232,"threads_started":1}
I20260811 22:26:02.207573 9854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:02.190369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:46573 (local address 127.29.106.1:60689)
0811 22:26:02.191084 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:02.191123 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:02.192835 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:02.193577 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:02.193587 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:02.194205 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:02.194756 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:02.194774 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:02.195971 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:02.195974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:02.197190 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:02.197203 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:02.197970 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:02.197976 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:02.198204 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:02.198842 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:02.198864 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:02.201003 (+ 2139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:26:02.203828 (+ 2825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:02.203837 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:02.203853 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:02.204264 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:02.204617 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:02.204622 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:02.204626 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:02.204804 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:02.205315 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:26:02.205324 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:26:02.205626 (+ 302us) client_negotiation.cc:769] Sending connection context
0811 22:26:02.205859 (+ 233us) client_negotiation.cc:240] Negotiation successful
0811 22:26:02.206364 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":278,"threads_started":1}
I20260811 22:26:02.209080 9850 heartbeater.cc:347] Connected to a master server at 127.29.106.62:46573
I20260811 22:26:02.209426 9850 heartbeater.cc:464] Registering TS with master...
I20260811 22:26:02.210268 9850 heartbeater.cc:511] Master 127.29.106.62:46573 requested a full tablet report, sending...
I20260811 22:26:02.212690 9487 ts_manager.cc:194] Registered new tserver with Master: f16cafe5e19446e782da1ed0fcd4a964 (127.29.106.1:35211)
I20260811 22:26:02.214726 9487 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.106.1@KRBTEST.COM'} at 127.29.106.1:60689
WARNING: no policy specified for oryx/127.29.106.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.106.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.106.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.2.keytab.
Entry for principal oryx/127.29.106.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.2.keytab.
I20260811 22:26:02.246763 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.2:0
--local_ip_for_outbound_sockets=127.29.106.2
--webserver_interface=127.29.106.2
--webserver_port=0
--tserver_master_addrs=127.29.106.62:46573
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.2.keytab
--principal=oryx/127.29.106.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:34763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:26:02.568778 9862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:26:02.569032 9862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:26:02.569154 9862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:26:02.576232 9862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:26:02.576325 9862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:26:02.576465 9862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.2
I20260811 22:26:02.585729 9862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:34763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.2.keytab
--principal=oryx/127.29.106.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.106.2
--webserver_port=0
--tserver_master_addrs=127.29.106.62:46573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.2
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:26:02.586997 9862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:26:02.588398 9862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:26:02.600827 9873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:02.600881 9872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:02.600857 9877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:26:02.601583 9862 server_base.cc:1034] running on GCE node
I20260811 22:26:02.602110 9862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:26:02.602914 9862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:26:02.604144 9862 hybrid_clock.cc:648] HybridClock initialized: now 1786487162604087 us; error 41 us; skew 500 ppm
I20260811 22:26:02.604449 9862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:26:02 dist-test-slave-gjrz krb5kdc[9387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487162, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.106.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:26:02.607503 9862 init.cc:377] Logged in from keytab as oryx/127.29.106.2@KRBTEST.COM (short username oryx)
I20260811 22:26:02.608858 9862 webserver.cc:460] Webserver started at http://127.29.106.2:42143/ using document root <none> and password file <none>
I20260811 22:26:02.609478 9862 fs_manager.cc:359] Metadata directory not provided
I20260811 22:26:02.609596 9862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:26:02.609870 9862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:26:02.612691 9862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-1/data/instance:
uuid: "7ff22bba56ee42879d1d18e2630d0fc3"
format_stamp: "Formatted at 2026-08-11 22:26:02 on dist-test-slave-gjrz"
server_key: "e6bd54a9cfb8b483ec8fafe77221ad96"
server_key_iv: "9b4c7b8ffe4f4009cfb490740e8ee3c6"
server_key_version: "encryptionkey@0"
I20260811 22:26:02.613399 9862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance:
uuid: "7ff22bba56ee42879d1d18e2630d0fc3"
format_stamp: "Formatted at 2026-08-11 22:26:02 on dist-test-slave-gjrz"
server_key: "e6bd54a9cfb8b483ec8fafe77221ad96"
server_key_iv: "9b4c7b8ffe4f4009cfb490740e8ee3c6"
server_key_version: "encryptionkey@0"
I20260811 22:26:02.618201 9862 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 22:26:02.621438 9887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:02.622670 9862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260811 22:26:02.622803 9862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-1/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-1/wal
uuid: "7ff22bba56ee42879d1d18e2630d0fc3"
format_stamp: "Formatted at 2026-08-11 22:26:02 on dist-test-slave-gjrz"
server_key: "e6bd54a9cfb8b483ec8fafe77221ad96"
server_key_iv: "9b4c7b8ffe4f4009cfb490740e8ee3c6"
server_key_version: "encryptionkey@0"
I20260811 22:26:02.622978 9862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:26:02.640467 9862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:26:02.644186 9862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:26:02.644440 9862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:26:02.645336 9862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:26:02.646672 9862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:26:02.646744 9862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:02.646855 9862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:26:02.646905 9862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:02.677289 9862 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.2:39773
I20260811 22:26:02.677363 10111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.2:39773 every 8 connection(s)
I20260811 22:26:02.678544 9862 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
I20260811 22:26:02.686774 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 9862
I20260811 22:26:02.686959 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance
I20260811 22:26:02.687415 30120 external_mini_cluster.cc:1428] Setting key cc977e83e5929ea9c6a585cd580b87bc
Aug 11 22:26:02 dist-test-slave-gjrz krb5kdc[9387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487162, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.106.2@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM
I20260811 22:26:02.698983 9853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:02.681119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:51773 (local address 127.29.106.62:46573)
0811 22:26:02.681331 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:02.681337 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:02.682519 (+ 1182us) server_negotiation.cc:407] Connection header received
0811 22:26:02.684593 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:02.684598 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:02.684681 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:02.684773 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:02.686271 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:02.687080 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:02.688887 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:02.689568 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:02.689692 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:02.693365 (+ 3673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:02.693370 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:02.693373 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:02.693405 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:02.695920 (+ 2515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:02.696797 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:02.696801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:02.696803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:02.696934 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:02.697379 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:02.697382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:02.697384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:02.697647 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:02.697757 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:02.698557 (+ 800us) server_negotiation.cc:299] Negotiation successful
0811 22:26:02.698747 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260811 22:26:02.700184 10115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:02.681574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:46573 (local address 127.29.106.2:51773)
0811 22:26:02.682343 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:02.682388 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:02.684299 (+ 1911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:02.685027 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:02.685035 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:02.685547 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:02.686094 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:02.686111 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:02.687235 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:02.687239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:02.688726 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:02.688742 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:02.689823 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:02.689831 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:02.690053 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:02.690707 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:02.690734 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:02.693116 (+ 2382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:26:02.696057 (+ 2941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:02.696066 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:02.696081 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:02.696663 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:02.697035 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:02.697039 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:02.697042 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:02.697281 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:02.697763 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:26:02.697771 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:26:02.698175 (+ 404us) client_negotiation.cc:769] Sending connection context
0811 22:26:02.698463 (+ 288us) client_negotiation.cc:240] Negotiation successful
0811 22:26:02.699011 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":317,"threads_started":1}
I20260811 22:26:02.701797 10112 heartbeater.cc:347] Connected to a master server at 127.29.106.62:46573
I20260811 22:26:02.702099 10112 heartbeater.cc:464] Registering TS with master...
I20260811 22:26:02.702836 10112 heartbeater.cc:511] Master 127.29.106.62:46573 requested a full tablet report, sending...
I20260811 22:26:02.704598 9487 ts_manager.cc:194] Registered new tserver with Master: 7ff22bba56ee42879d1d18e2630d0fc3 (127.29.106.2:39773)
I20260811 22:26:02.705681 9487 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.106.2@KRBTEST.COM'} at 127.29.106.2:51773
WARNING: no policy specified for oryx/127.29.106.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.106.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.106.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.3.keytab.
Entry for principal oryx/127.29.106.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.3.keytab.
I20260811 22:26:02.743319 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.3:0
--local_ip_for_outbound_sockets=127.29.106.3
--webserver_interface=127.29.106.3
--webserver_port=0
--tserver_master_addrs=127.29.106.62:46573
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.3.keytab
--principal=oryx/127.29.106.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:34763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:26:03.066547 10121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:26:03.066843 10121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:26:03.066948 10121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:26:03.074100 10121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:26:03.074208 10121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:26:03.074319 10121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.3
I20260811 22:26:03.084462 10121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:34763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.3.keytab
--principal=oryx/127.29.106.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.106.3
--webserver_port=0
--tserver_master_addrs=127.29.106.62:46573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.3
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:26:03.085588 10121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:26:03.086792 10121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:26:03.099315 10139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:03.099345 10147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:03.099496 10150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:26:03.100087 10121 server_base.cc:1034] running on GCE node
I20260811 22:26:03.100608 10121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:26:03.101439 10121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:26:03.102632 10121 hybrid_clock.cc:648] HybridClock initialized: now 1786487163102591 us; error 31 us; skew 500 ppm
I20260811 22:26:03.102924 10121 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:26:03 dist-test-slave-gjrz krb5kdc[9387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487163, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.106.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:26:03.105978 10121 init.cc:377] Logged in from keytab as oryx/127.29.106.3@KRBTEST.COM (short username oryx)
I20260811 22:26:03.107522 10121 webserver.cc:460] Webserver started at http://127.29.106.3:43601/ using document root <none> and password file <none>
I20260811 22:26:03.108207 10121 fs_manager.cc:359] Metadata directory not provided
I20260811 22:26:03.108330 10121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:26:03.108642 10121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:26:03.111928 10121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-2/data/instance:
uuid: "5828483962134e1b972872a8dd583692"
format_stamp: "Formatted at 2026-08-11 22:26:03 on dist-test-slave-gjrz"
server_key: "a48d02f00265cd64e241955f8acf635f"
server_key_iv: "7f0148234c7c86b59fdae66ed9e19f8a"
server_key_version: "encryptionkey@0"
I20260811 22:26:03.112749 10121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance:
uuid: "5828483962134e1b972872a8dd583692"
format_stamp: "Formatted at 2026-08-11 22:26:03 on dist-test-slave-gjrz"
server_key: "a48d02f00265cd64e241955f8acf635f"
server_key_iv: "7f0148234c7c86b59fdae66ed9e19f8a"
server_key_version: "encryptionkey@0"
I20260811 22:26:03.118067 10121 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260811 22:26:03.121472 10158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:03.122792 10121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 22:26:03.122953 10121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-2/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-2/wal
uuid: "5828483962134e1b972872a8dd583692"
format_stamp: "Formatted at 2026-08-11 22:26:03 on dist-test-slave-gjrz"
server_key: "a48d02f00265cd64e241955f8acf635f"
server_key_iv: "7f0148234c7c86b59fdae66ed9e19f8a"
server_key_version: "encryptionkey@0"
I20260811 22:26:03.123134 10121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:26:03.142007 10121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:26:03.145800 10121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:26:03.146080 10121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:26:03.147020 10121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:26:03.148418 10121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:26:03.148499 10121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:03.148624 10121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:26:03.148680 10121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:03.179617 10121 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.3:38811
I20260811 22:26:03.179683 10372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.3:38811 every 8 connection(s)
I20260811 22:26:03.180931 10121 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
I20260811 22:26:03.185984 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 10121
I20260811 22:26:03.186195 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance
I20260811 22:26:03.186734 30120 external_mini_cluster.cc:1428] Setting key 8ea728da284fe74ec86bbf75a0e54975
Aug 11 22:26:03 dist-test-slave-gjrz krb5kdc[9387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487163, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.106.3@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM
I20260811 22:26:03.201683 9853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:03.183505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:43255 (local address 127.29.106.62:46573)
0811 22:26:03.183718 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:03.183725 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:03.184998 (+ 1273us) server_negotiation.cc:407] Connection header received
0811 22:26:03.187501 (+ 2503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:03.187508 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:03.187600 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:03.187705 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:03.189745 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:03.190597 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:03.192232 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:03.192639 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:03.192739 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:03.196031 (+ 3292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:03.196036 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:03.196054 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:03.196086 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:03.198609 (+ 2523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:03.199333 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:03.199338 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:03.199342 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:03.199468 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:03.199891 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:03.199895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:03.199897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:03.200166 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:03.200298 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:03.201120 (+ 822us) server_negotiation.cc:299] Negotiation successful
0811 22:26:03.201342 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260811 22:26:03.202668 10378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:03.184080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:46573 (local address 127.29.106.3:43255)
0811 22:26:03.184807 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:03.184853 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:03.187117 (+ 2264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:03.188001 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:03.188014 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:03.188772 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:03.189542 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:03.189574 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:03.190765 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:03.190770 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:03.192083 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:03.192095 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:03.192860 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:03.192866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:03.193066 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:03.193662 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:03.193688 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:03.195827 (+ 2139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:26:03.198746 (+ 2919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:03.198756 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:03.198771 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:03.199187 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:03.199579 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:03.199585 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:03.199589 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:03.199770 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:03.200293 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:26:03.200301 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:26:03.200665 (+ 364us) client_negotiation.cc:769] Sending connection context
0811 22:26:03.200963 (+ 298us) client_negotiation.cc:240] Negotiation successful
0811 22:26:03.201467 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":285,"threads_started":1}
I20260811 22:26:03.203891 10373 heartbeater.cc:347] Connected to a master server at 127.29.106.62:46573
I20260811 22:26:03.204146 10373 heartbeater.cc:464] Registering TS with master...
I20260811 22:26:03.204824 10373 heartbeater.cc:511] Master 127.29.106.62:46573 requested a full tablet report, sending...
I20260811 22:26:03.206162 9487 ts_manager.cc:194] Registered new tserver with Master: 5828483962134e1b972872a8dd583692 (127.29.106.3:38811)
I20260811 22:26:03.206954 9487 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.106.3@KRBTEST.COM'} at 127.29.106.3:43255
I20260811 22:26:03.214588 30120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 22:26:03.218281 9850 heartbeater.cc:503] Master 127.29.106.62:46573 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.29.106.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.106.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.106.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.61.keytab.
Entry for principal oryx/127.29.106.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.61.keytab.
I20260811 22:26:03.266170 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.61:46755
--webserver_interface=127.29.106.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.61.keytab
--principal=oryx/127.29.106.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:34763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:46573,127.29.106.61:46755
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:26:03.592587 10382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:26:03.592871 10382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:26:03.592962 10382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:26:03.599977 10382 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:26:03.600075 10382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:26:03.600131 10382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:26:03.600186 10382 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:26:03.600230 10382 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:26:03.609735 10382 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:34763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:46573,127.29.106.61:46755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.61.keytab
--principal=oryx/127.29.106.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.61:46755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.29.106.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
W20260811 22:26:03.610044 10382 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.29.106.62:46573,127.29.106.61:46755'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260811 22:26:03.611023 10382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:26:03.612309 10382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:26:03.623514 10392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:03.623474 10395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:03.623546 10390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:26:03.624776 10382 server_base.cc:1034] running on GCE node
I20260811 22:26:03.625550 10382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:26:03.627218 10382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:26:03.628485 10382 hybrid_clock.cc:648] HybridClock initialized: now 1786487163628361 us; error 96 us; skew 500 ppm
I20260811 22:26:03.628830 10382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:26:03 dist-test-slave-gjrz krb5kdc[9387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487163, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.106.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:26:03.632061 10382 init.cc:377] Logged in from keytab as oryx/127.29.106.61@KRBTEST.COM (short username oryx)
I20260811 22:26:03.633502 10382 webserver.cc:460] Webserver started at http://127.29.106.61:40905/ using document root <none> and password file <none>
I20260811 22:26:03.634163 10382 fs_manager.cc:359] Metadata directory not provided
I20260811 22:26:03.634253 10382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:26:03.634508 10382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:26:03.637418 10382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-1/data/instance:
uuid: "e0ad6642bf744796b25dfefd04d3f245"
format_stamp: "Formatted at 2026-08-11 22:26:03 on dist-test-slave-gjrz"
server_key: "b5b0deaec3e04514b5a55d7b3bd35619"
server_key_iv: "1f8fc73d44ffeea3971d91eb150fd3f9"
server_key_version: "encryptionkey@0"
I20260811 22:26:03.638193 10382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-1/wal/instance:
uuid: "e0ad6642bf744796b25dfefd04d3f245"
format_stamp: "Formatted at 2026-08-11 22:26:03 on dist-test-slave-gjrz"
server_key: "b5b0deaec3e04514b5a55d7b3bd35619"
server_key_iv: "1f8fc73d44ffeea3971d91eb150fd3f9"
server_key_version: "encryptionkey@0"
I20260811 22:26:03.643049 10382 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260811 22:26:03.646333 10403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:03.647728 10382 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:26:03.647894 10382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-1/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-1/wal
uuid: "e0ad6642bf744796b25dfefd04d3f245"
format_stamp: "Formatted at 2026-08-11 22:26:03 on dist-test-slave-gjrz"
server_key: "b5b0deaec3e04514b5a55d7b3bd35619"
server_key_iv: "1f8fc73d44ffeea3971d91eb150fd3f9"
server_key_version: "encryptionkey@0"
I20260811 22:26:03.648082 10382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:26:03.665078 10382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:26:03.669229 10382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:26:03.669445 10382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:26:03.687934 10382 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.61:46755
I20260811 22:26:03.687958 10504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.61:46755 every 8 connection(s)
I20260811 22:26:03.689263 10382 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-1/data/info.pb
I20260811 22:26:03.693022 10505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:26:03.698913 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 10382
I20260811 22:26:03.699455 10505 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } has no permanent_uuid. Determining permanent_uuid...
I20260811 22:26:03.708811 10112 heartbeater.cc:503] Master 127.29.106.62:46573 was elected leader, sending a full tablet report...
Aug 11 22:26:03 dist-test-slave-gjrz krb5kdc[9387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487163, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.106.61@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM
Aug 11 22:26:03 dist-test-slave-gjrz krb5kdc[9387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.106.61@KRBTEST.COM
I20260811 22:26:03.718902 10513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:03.702283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42326 (local address 127.29.106.62:46573)
0811 22:26:03.702721 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:03.702727 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:03.703398 (+ 671us) server_negotiation.cc:407] Connection header received
0811 22:26:03.704321 (+ 923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:03.704327 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:03.704396 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:03.704492 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:03.705766 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:03.706590 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:03.708324 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:03.708825 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:03.708954 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:03.712904 (+ 3950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:03.712908 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:03.712911 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:03.712945 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:03.715594 (+ 2649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:03.716554 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:03.716558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:03.716561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:03.716688 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:03.717133 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:03.717136 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:03.717138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:03.717528 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:03.717668 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:03.718362 (+ 694us) server_negotiation.cc:299] Negotiation successful
0811 22:26:03.718582 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":225,"threads_started":1}
I20260811 22:26:03.720079 10514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:03.702347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:46573 (local address 127.0.0.1:42326)
0811 22:26:03.703110 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:03.703171 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:03.704061 (+ 890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:03.704587 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:03.704598 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:03.705207 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:03.705590 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:03.705607 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:03.706777 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:03.706781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:03.708179 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:03.708194 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:03.709164 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:03.709172 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:03.709376 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:03.710246 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:03.710268 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:03.712719 (+ 2451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:26:03.715719 (+ 3000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:03.715732 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:03.715745 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:03.716439 (+ 694us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:03.716797 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:03.716801 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:03.716805 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:03.717034 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:03.717659 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:26:03.717667 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:26:03.717977 (+ 310us) client_negotiation.cc:769] Sending connection context
0811 22:26:03.718225 (+ 248us) client_negotiation.cc:240] Negotiation successful
0811 22:26:03.718696 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"mutex_wait_us":55,"thread_start_us":399,"threads_started":1}
I20260811 22:26:03.720105 10512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:03.701030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56726 (local address 127.29.106.61:46755)
0811 22:26:03.701756 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:03.701770 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:03.701816 (+ 46us) server_negotiation.cc:407] Connection header received
0811 22:26:03.704056 (+ 2240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:03.704078 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:03.704306 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:03.704481 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:03.706008 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:03.707248 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:03.708638 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:03.709313 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:03.709424 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:03.714267 (+ 4843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:03.714275 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:03.714287 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:03.714319 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:03.717413 (+ 3094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:03.718035 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:03.718044 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:03.718048 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:03.718187 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:03.718545 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:03.718550 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:03.718554 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:03.718874 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:03.719144 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:03.719532 (+ 388us) server_negotiation.cc:299] Negotiation successful
0811 22:26:03.719755 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":537,"spinlock_wait_cycles":26880,"thread_start_us":330,"threads_started":1}
I20260811 22:26:03.721983 10505 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.106.61" port: 46755 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 22:26:03 dist-test-slave-gjrz krb5kdc[9387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487163, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.106.61@KRBTEST.COM for oryx/127.29.106.61@KRBTEST.COM
I20260811 22:26:03.732969 10514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:03.722784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.61:46755 (local address 127.0.0.1:56732)
0811 22:26:03.723011 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:03.723024 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:03.723112 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:03.723351 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:03.723355 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:03.723637 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:03.723756 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:03.723763 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:03.724627 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:03.724630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:03.725463 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:03.725473 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:03.726125 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:03.726130 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:03.726287 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:03.726828 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:03.726843 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:03.728716 (+ 1873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:26:03.731099 (+ 2383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:03.731102 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:03.731105 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:03.731503 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:03.731798 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:03.731801 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:03.731804 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:03.731924 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:03.732343 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:26:03.732346 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:26:03.732452 (+ 106us) client_negotiation.cc:769] Sending connection context
0811 22:26:03.732555 (+ 103us) client_negotiation.cc:240] Negotiation successful
0811 22:26:03.732734 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176}
I20260811 22:26:03.733094 10512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:03.722777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56732 (local address 127.29.106.61:46755)
0811 22:26:03.722945 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:03.722949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:03.723095 (+ 146us) server_negotiation.cc:407] Connection header received
0811 22:26:03.723186 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:03.723190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:03.723240 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:03.723340 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:03.723862 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:03.724509 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:03.725591 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:03.725996 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:03.726114 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:03.728868 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:03.728872 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:03.728875 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:03.728901 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:03.730987 (+ 2086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:03.731606 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:03.731610 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:03.731612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:03.731722 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:03.732012 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:03.732015 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:03.732017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:03.732254 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:03.732353 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:03.732621 (+ 268us) server_negotiation.cc:299] Negotiation successful
0811 22:26:03.732826 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":58}
I20260811 22:26:03.737416 10505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245: Bootstrap starting.
I20260811 22:26:03.740381 10505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245: Neither blocks nor log segments found. Creating new log.
I20260811 22:26:03.741348 10505 log.cc:824] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245: Log is configured to *not* fsync() on all Append() calls
I20260811 22:26:03.743839 10505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245: No bootstrap required, opened a new log
I20260811 22:26:03.750802 10505 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 46755 } }
I20260811 22:26:03.751183 10505 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:26:03.751318 10505 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0ad6642bf744796b25dfefd04d3f245, State: Initialized, Role: FOLLOWER
I20260811 22:26:03.751914 10505 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 46755 } }
I20260811 22:26:03.752803 10520 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 46755 } } }
I20260811 22:26:03.753124 10520 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:26:03.753770 10505 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:26:03.757234 10520 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:26:03.757539 10520 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 46755 } }
I20260811 22:26:03.758716 10520 leader_election.cc:288] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b5d1ed0b161417688f43a878b41ef3d (127.29.106.62:46573)
I20260811 22:26:03.759814 9503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e0ad6642bf744796b25dfefd04d3f245" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b5d1ed0b161417688f43a878b41ef3d" is_pre_election: true
I20260811 22:26:03.761415 10405 leader_election.cc:302] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: e0ad6642bf744796b25dfefd04d3f245; no voters: 2b5d1ed0b161417688f43a878b41ef3d
I20260811 22:26:03.761830 10520 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260811 22:26:03.763391 10542 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 22:26:03.763538 10542 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 22:26:03.766544 10382 master_runner.cc:428] Detected that this master e0ad6642bf744796b25dfefd04d3f245 is joining an existing cluster
I20260811 22:26:03.766625 10382 master_runner.cc:432] Initiating AddMaster RPC to add 127.29.106.61:46755
I20260811 22:26:03.785414 10512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:03.770598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56738 (local address 127.29.106.61:46755)
0811 22:26:03.770776 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:03.770782 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:03.770976 (+ 194us) server_negotiation.cc:407] Connection header received
0811 22:26:03.771230 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:03.771235 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:03.771331 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:03.771438 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:03.772995 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:03.773784 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:03.775593 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:03.776353 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:03.776499 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:03.778623 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:03.778630 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:03.778635 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:03.778669 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:03.782680 (+ 4011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:03.783414 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:03.783419 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:03.783421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:03.783562 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:03.783915 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:03.783919 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:03.783921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:03.784277 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:03.784393 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:03.784839 (+ 446us) server_negotiation.cc:299] Negotiation successful
0811 22:26:03.785061 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":50}
I20260811 22:26:03.786263 10551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:03.770174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.61:46755 (local address 127.0.0.1:56738)
0811 22:26:03.770878 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:03.770899 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:03.771086 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:03.771640 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:03.771644 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:03.772431 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:03.772848 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:03.772861 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:03.773957 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:03.773962 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:03.775456 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:03.775470 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:03.776551 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:03.776557 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:03.776736 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:03.777431 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:03.777457 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:03.778447 (+ 990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:26:03.782827 (+ 4380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:03.782831 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:03.782835 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:03.783312 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:03.783677 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:03.783680 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:03.783683 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:03.783822 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:03.784469 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:26:03.784472 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:26:03.784621 (+ 149us) client_negotiation.cc:769] Sending connection context
0811 22:26:03.785592 (+ 971us) client_negotiation.cc:240] Negotiation successful
0811 22:26:03.785869 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"spinlock_wait_cycles":211200,"thread_start_us":606,"threads_started":1}
I20260811 22:26:03.788524 10553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:03.770257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:46573 (local address 127.0.0.1:42336)
0811 22:26:03.771726 (+ 1469us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:03.771745 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:03.771852 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:03.772200 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:03.772205 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:03.772785 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:03.773107 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:03.773119 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:03.774573 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:03.774582 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:03.777198 (+ 2616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:03.777212 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:03.778400 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:03.778407 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:03.778600 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:03.779200 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:03.779223 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:03.780438 (+ 1215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:26:03.784322 (+ 3884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:03.784327 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:03.784330 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:03.784834 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:03.785285 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:03.785290 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:03.785294 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:03.785834 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:03.787312 (+ 1478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:26:03.787318 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:26:03.787536 (+ 218us) client_negotiation.cc:769] Sending connection context
0811 22:26:03.787678 (+ 142us) client_negotiation.cc:240] Negotiation successful
0811 22:26:03.787987 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1356,"mutex_wait_us":1,"thread_start_us":1189,"threads_started":1}
I20260811 22:26:03.789559 10513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:03.769979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42336 (local address 127.29.106.62:46573)
0811 22:26:03.770153 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:03.770159 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:03.771828 (+ 1669us) server_negotiation.cc:407] Connection header received
0811 22:26:03.771974 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:03.771979 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:03.772073 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:03.772156 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:03.773274 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:03.774360 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:03.777358 (+ 2998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:03.778015 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:03.778114 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:03.780659 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:03.780665 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:03.780670 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:03.780709 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:03.784167 (+ 3458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:03.784974 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:03.784980 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:03.784984 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:03.785143 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:03.785974 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:03.785980 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:03.785983 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:03.787106 (+ 1123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:03.788915 (+ 1809us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:03.789021 (+ 106us) server_negotiation.cc:299] Negotiation successful
0811 22:26:03.789327 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
WARNING: no policy specified for oryx/127.29.106.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.106.60@KRBTEST.COM" created.
Aug 11 22:26:03 dist-test-slave-gjrz krb5kdc[9387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487161, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.106.62@KRBTEST.COM for oryx/127.29.106.61@KRBTEST.COM
I20260811 22:26:03.815227 10512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:03.796891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56744 (local address 127.29.106.61:46755)
0811 22:26:03.797658 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:03.797664 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:03.797681 (+ 17us) server_negotiation.cc:407] Connection header received
0811 22:26:03.797763 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:03.797768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:03.797849 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:03.797947 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:03.799348 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:03.800482 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:03.802394 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:03.803129 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:03.803300 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:03.808450 (+ 5150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:03.808457 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:03.808462 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:03.808503 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:03.812093 (+ 3590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:03.812969 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:03.812975 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:03.812978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:03.813133 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:03.813565 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:03.813569 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:03.813572 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:03.813912 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:03.814088 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:03.814655 (+ 567us) server_negotiation.cc:299] Negotiation successful
0811 22:26:03.814872 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644}
I20260811 22:26:03.815317 10555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:03.796626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.61:46755 (local address 127.0.0.1:56744)
0811 22:26:03.797246 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:03.797286 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:03.797439 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:03.798040 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:03.798047 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:03.798857 (+ 810us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:03.799157 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:03.799170 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:03.800646 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:03.800651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:03.802223 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:03.802240 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:03.803355 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:03.803363 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:03.803598 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:03.804315 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:03.804332 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:03.808238 (+ 3906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:26:03.812254 (+ 4016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:03.812267 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:03.812275 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:03.812831 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:03.813262 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:03.813267 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:03.813271 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:03.813449 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:03.814092 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:26:03.814101 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:26:03.814303 (+ 202us) client_negotiation.cc:769] Sending connection context
0811 22:26:03.814516 (+ 213us) client_negotiation.cc:240] Negotiation successful
0811 22:26:03.814860 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":313,"threads_started":1}
I20260811 22:26:03.817644 9487 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.29.106.61:46755: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245"
member_type: NON_VOTER
last_known_addr {
host: "127.29.106.61"
port: 46755
}
attrs {
promote: true
}
}
dest_uuid: "2b5d1ed0b161417688f43a878b41ef3d"
cas_config_opid_index: -1
I20260811 22:26:03.819190 9487 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: NON_VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: true } }
I20260811 22:26:03.831691 10560 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER e0ad6642bf744796b25dfefd04d3f245 (127.29.106.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: NON_VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: true } } }
I20260811 22:26:03.833263 10455 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:26:03.836508 10560 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.29.106.61:46755
I20260811 22:26:03.837080 10560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2b5d1ed0b161417688f43a878b41ef3d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: NON_VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: true } } }
I20260811 22:26:03.837323 10560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d [sys.catalog]: This master's current role is: LEADER
I20260811 22:26:03.837383 10455 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245 [term 1 FOLLOWER]: Refusing update from remote peer 2b5d1ed0b161417688f43a878b41ef3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260811 22:26:03.837771 10382 master.cc:537] Master@127.29.106.61:46755 shutting down...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.106.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.60.keytab.
Entry for principal oryx/127.29.106.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.60.keytab.
I20260811 22:26:03.840394 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.60:40553
--webserver_interface=127.29.106.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.60.keytab
--principal=oryx/127.29.106.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:34763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:46573,127.29.106.61:46755,127.29.106.60:40553
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 22:26:03.840521 10560 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: NON_VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260811 22:26:03.843834 9455 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260811 22:26:03.847954 9455 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d -> Peer e0ad6642bf744796b25dfefd04d3f245 (127.29.106.61:46755): Couldn't send request to peer e0ad6642bf744796b25dfefd04d3f245. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260811 22:26:03.852422 10382 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:26:03.852797 10382 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:26:03.852968 10382 tablet_replica.cc:330] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245: stopping tablet replica
I20260811 22:26:03.859323 10382 master.cc:552] Master@127.29.106.61:46755 shutdown complete.
I20260811 22:26:03.859445 10382 master_runner.cc:305] Clearing existing system tablet
I20260811 22:26:03.859838 10382 fs_manager.cc:359] Metadata directory not provided
I20260811 22:26:03.859929 10382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:26:03.862145 10382 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 22:26:03.863546 10572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:03.863811 10382 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:26:03.864089 10382 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 22:26:03.864180 10382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-1/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-1/wal
uuid: "e0ad6642bf744796b25dfefd04d3f245"
format_stamp: "Formatted at 2026-08-11 22:26:03 on dist-test-slave-gjrz"
server_key: "b5b0deaec3e04514b5a55d7b3bd35619"
server_key_iv: "1f8fc73d44ffeea3971d91eb150fd3f9"
server_key_version: "encryptionkey@0"
I20260811 22:26:03.868551 10382 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:26:03.873643 10382 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260811 22:26:03.873808 10382 log.cc:1196] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260811 22:26:03.874217 10382 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245: Deleting consensus metadata
I20260811 22:26:03.874571 10382 master_runner.cc:315] Copying system tablet from 127.29.106.62:46573
I20260811 22:26:03.877457 10382 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245: tablet copy: Beginning tablet copy session from remote peer at address 127.29.106.62:46573
I20260811 22:26:03.890441 10513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:03.878410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42340 (local address 127.29.106.62:46573)
0811 22:26:03.878631 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:03.878638 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:03.879809 (+ 1171us) server_negotiation.cc:407] Connection header received
0811 22:26:03.880000 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:03.880005 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:03.880072 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:03.880158 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:03.880987 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:03.881721 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:03.882991 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:03.883609 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:03.883718 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:03.885452 (+ 1734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:03.885457 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:03.885460 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:03.885492 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:03.887963 (+ 2471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:03.888564 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:03.888569 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:03.888572 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:03.888695 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:03.889054 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:03.889059 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:03.889063 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:03.889330 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:03.889465 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:03.889868 (+ 403us) server_negotiation.cc:299] Negotiation successful
0811 22:26:03.890091 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260811 22:26:03.890501 10587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:03.878346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:46573 (local address 127.0.0.1:42340)
0811 22:26:03.879694 (+ 1348us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:03.879711 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:03.879855 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:03.880227 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:03.880231 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:03.880620 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:03.880831 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:03.880840 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:03.881882 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:03.881887 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:03.882827 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:03.882839 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:03.883758 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:03.883764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:03.883951 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:03.884535 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:03.884557 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:03.885263 (+ 706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:26:03.888093 (+ 2830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:03.888099 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:03.888103 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:03.888451 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:03.888804 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:03.888809 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:03.888812 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:03.888951 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:03.889456 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:26:03.889460 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:26:03.889604 (+ 144us) client_negotiation.cc:769] Sending connection context
0811 22:26:03.889738 (+ 134us) client_negotiation.cc:240] Negotiation successful
0811 22:26:03.890036 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1236,"thread_start_us":1060,"threads_started":1}
I20260811 22:26:03.891723 9520 tablet_copy_service.cc:140] P 2b5d1ed0b161417688f43a878b41ef3d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e0ad6642bf744796b25dfefd04d3f245 ({username='oryx', principal='oryx/127.29.106.61@KRBTEST.COM'} at 127.0.0.1:42340)
I20260811 22:26:03.892069 9520 tablet_copy_service.cc:161] P 2b5d1ed0b161417688f43a878b41ef3d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e0ad6642bf744796b25dfefd04d3f245 at {username='oryx', principal='oryx/127.29.106.61@KRBTEST.COM'} at 127.0.0.1:42340: session id = e0ad6642bf744796b25dfefd04d3f245-00000000000000000000000000000000
I20260811 22:26:03.895722 9520 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d: Tablet Copy: opened 0 blocks and 1 log segments
I20260811 22:26:03.897874 10382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260811 22:26:03.902416 10382 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245: tablet copy: Starting download of 0 data blocks...
I20260811 22:26:03.902733 10382 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245: tablet copy: Starting download of 1 WAL segments...
I20260811 22:26:03.905772 10382 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260811 22:26:03.909380 9520 tablet_copy_service.cc:342] P 2b5d1ed0b161417688f43a878b41ef3d: Request end of tablet copy session e0ad6642bf744796b25dfefd04d3f245-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.29.106.61@KRBTEST.COM'} at 127.0.0.1:42340
I20260811 22:26:03.909615 9520 tablet_copy_service.cc:433] P 2b5d1ed0b161417688f43a878b41ef3d: ending tablet copy session e0ad6642bf744796b25dfefd04d3f245-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e0ad6642bf744796b25dfefd04d3f245
I20260811 22:26:03.920924 10382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:26:03.923405 10588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:03.924928 10589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:03.940747 10591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:26:03.941535 10382 server_base.cc:1034] running on GCE node
I20260811 22:26:03.941960 10382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:26:03.942734 10382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:26:03.943958 10382 hybrid_clock.cc:648] HybridClock initialized: now 1786487163943878 us; error 42 us; skew 500 ppm
I20260811 22:26:03.944185 10382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:26:03 dist-test-slave-gjrz krb5kdc[9387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487163, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.106.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:26:03.946419 10382 init.cc:377] Logged in from keytab as oryx/127.29.106.61@KRBTEST.COM (short username oryx)
I20260811 22:26:03.952255 10382 webserver.cc:460] Webserver started at http://127.29.106.61:46703/ using document root <none> and password file <none>
I20260811 22:26:03.952684 10382 fs_manager.cc:359] Metadata directory not provided
I20260811 22:26:03.952783 10382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:26:03.956106 10382 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:26:03.958094 10608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:03.958530 10382 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 22:26:03.958703 10382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-1/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-1/wal
uuid: "e0ad6642bf744796b25dfefd04d3f245"
format_stamp: "Formatted at 2026-08-11 22:26:03 on dist-test-slave-gjrz"
server_key: "b5b0deaec3e04514b5a55d7b3bd35619"
server_key_iv: "1f8fc73d44ffeea3971d91eb150fd3f9"
server_key_version: "encryptionkey@0"
I20260811 22:26:03.958950 10382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:26:03.992153 10382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:26:03.996824 10382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:26:04.019574 10382 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.61:46755
I20260811 22:26:04.019601 10716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.61:46755 every 8 connection(s)
I20260811 22:26:04.020720 10382 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-1/data/info.pb
I20260811 22:26:04.026002 10717 sys_catalog.cc:263] Verifying existing consensus state
I20260811 22:26:04.040726 10717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245: Bootstrap starting.
I20260811 22:26:04.063947 10717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 22:26:04.064772 10717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245: Bootstrap complete.
I20260811 22:26:04.066462 10717 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: NON_VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: true } }
I20260811 22:26:04.066875 10717 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e0ad6642bf744796b25dfefd04d3f245, State: Initialized, Role: LEARNER
I20260811 22:26:04.067257 10717 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: NON_VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: true } }
I20260811 22:26:04.068871 10727 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: NON_VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: true } } }
I20260811 22:26:04.069757 10727 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245 [sys.catalog]: This master's current role is: LEARNER
I20260811 22:26:04.069435 10717 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:26:04.117455 10746 catalog_manager.cc:1259] Loaded cluster ID: 8731d947535c4e8c8f3224f1af27f098
I20260811 22:26:04.117612 10746 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245: loading cluster ID for follower catalog manager: success
I20260811 22:26:04.124157 10746 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245: acquiring CA information for follower catalog manager: success
I20260811 22:26:04.128185 10746 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 22:26:04.209481 10373 heartbeater.cc:503] Master 127.29.106.62:46573 was elected leader, sending a full tablet report...
I20260811 22:26:04.450510 10756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:04.443134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56754 (local address 127.29.106.61:46755)
0811 22:26:04.444421 (+ 1287us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:04.444431 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:04.444478 (+ 47us) server_negotiation.cc:407] Connection header received
0811 22:26:04.444543 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:04.444547 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:04.444646 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:04.444749 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:26:04.445296 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:04.446285 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:04.448648 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:04.449502 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:04.449610 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:04.449799 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:04.450126 (+ 327us) server_negotiation.cc:299] Negotiation successful
0811 22:26:04.450243 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1140,"thread_start_us":920,"threads_started":1}
I20260811 22:26:04.450567 10755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:04.442844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.61:46755 (local address 127.0.0.1:56754)
0811 22:26:04.443701 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:04.443724 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:04.443885 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:04.444854 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:04.444860 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:04.444895 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:26:04.445113 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:04.445126 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:04.446471 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:04.446478 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:04.448485 (+ 2007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:04.448504 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:04.449705 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:04.449712 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:04.449895 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:04.449960 (+ 65us) client_negotiation.cc:769] Sending connection context
0811 22:26:04.450065 (+ 105us) client_negotiation.cc:240] Negotiation successful
0811 22:26:04.450213 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":600,"threads_started":1}
I20260811 22:26:04.451944 10669 raft_consensus.cc:1203] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260811 22:26:04.452651 10727 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b5d1ed0b161417688f43a878b41ef3d. Latest consensus state: current_term: 1 leader_uuid: "2b5d1ed0b161417688f43a878b41ef3d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: NON_VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: true } } }
I20260811 22:26:04.452944 10727 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245 [sys.catalog]: This master's current role is: LEARNER
W20260811 22:26:04.486227 10567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:26:04.486716 10567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:26:04.487010 10567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:26:04.501082 10567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:26:04.501302 10567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:26:04.501472 10567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:26:04.501684 10567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:26:04.501828 10567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:26:04.517081 10567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:34763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:46573,127.29.106.61:46755,127.29.106.60:40553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.60.keytab
--principal=oryx/127.29.106.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.60:40553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.29.106.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:26:04.518703 10567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:26:04.520138 10567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:26:04.538195 10762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:04.538275 10764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:04.538980 10766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:26:04.540732 10567 server_base.cc:1034] running on GCE node
I20260811 22:26:04.541491 10567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:26:04.542888 10567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:26:04.544174 10567 hybrid_clock.cc:648] HybridClock initialized: now 1786487164544052 us; error 40 us; skew 500 ppm
I20260811 22:26:04.544612 10567 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:26:04 dist-test-slave-gjrz krb5kdc[9387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487164, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.106.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:26:04.549708 10567 init.cc:377] Logged in from keytab as oryx/127.29.106.60@KRBTEST.COM (short username oryx)
I20260811 22:26:04.551985 10567 webserver.cc:460] Webserver started at http://127.29.106.60:44151/ using document root <none> and password file <none>
I20260811 22:26:04.553037 10567 fs_manager.cc:359] Metadata directory not provided
I20260811 22:26:04.553283 10567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:26:04.553690 10567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:26:04.557785 10567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-2/data/instance:
uuid: "800b0066b8d34ef98b9c2bc24409ee79"
format_stamp: "Formatted at 2026-08-11 22:26:04 on dist-test-slave-gjrz"
server_key: "cd3165ef52498f1aae0982fec35e6881"
server_key_iv: "97480168abee36db1789e58a0ff7b6be"
server_key_version: "encryptionkey@0"
I20260811 22:26:04.558732 10567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-2/wal/instance:
uuid: "800b0066b8d34ef98b9c2bc24409ee79"
format_stamp: "Formatted at 2026-08-11 22:26:04 on dist-test-slave-gjrz"
server_key: "cd3165ef52498f1aae0982fec35e6881"
server_key_iv: "97480168abee36db1789e58a0ff7b6be"
server_key_version: "encryptionkey@0"
I20260811 22:26:04.565608 10567 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260811 22:26:04.570705 10781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:04.572203 10567 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260811 22:26:04.572387 10567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-2/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-2/wal
uuid: "800b0066b8d34ef98b9c2bc24409ee79"
format_stamp: "Formatted at 2026-08-11 22:26:04 on dist-test-slave-gjrz"
server_key: "cd3165ef52498f1aae0982fec35e6881"
server_key_iv: "97480168abee36db1789e58a0ff7b6be"
server_key_version: "encryptionkey@0"
I20260811 22:26:04.572629 10567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:26:04.604848 10567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:26:04.611443 10567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:26:04.611976 10567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:26:04.667737 10567 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.60:40553
I20260811 22:26:04.667906 10888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.60:40553 every 8 connection(s)
I20260811 22:26:04.669631 10567 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-2/data/info.pb
I20260811 22:26:04.671450 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 10567
I20260811 22:26:04.676604 10892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:26:04.689407 10892 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 22:26:04 dist-test-slave-gjrz krb5kdc[9387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.106.60@KRBTEST.COM
Aug 11 22:26:04 dist-test-slave-gjrz krb5kdc[9387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487164, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.106.60@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM
I20260811 22:26:04.707561 10899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:04.674389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47090 (local address 127.29.106.60:40553)
0811 22:26:04.676220 (+ 1831us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:04.676235 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:04.676322 (+ 87us) server_negotiation.cc:407] Connection header received
0811 22:26:04.680251 (+ 3929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:04.680277 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:04.680803 (+ 526us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:04.681348 (+ 545us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:04.683615 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:04.686158 (+ 2543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:04.688159 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:04.689520 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:04.689706 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:04.697226 (+ 7520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:04.697240 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:04.697255 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:04.697338 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:04.702057 (+ 4719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:04.702888 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:04.702894 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:04.702901 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:04.703190 (+ 289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:04.703871 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:04.703876 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:04.703880 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:04.704580 (+ 700us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:04.704960 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:04.705456 (+ 496us) server_negotiation.cc:299] Negotiation successful
0811 22:26:04.705850 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1537,"thread_start_us":1053,"threads_started":1}
I20260811 22:26:04.716807 10902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:04.693271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:46573 (local address 127.0.0.1:42346)
0811 22:26:04.694978 (+ 1707us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:04.695015 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:04.695217 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:04.695700 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:04.695707 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:04.696957 (+ 1250us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:04.697888 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:04.697907 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:04.699392 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:04.699397 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:04.701322 (+ 1925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:04.701335 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:04.702558 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:04.702568 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:04.702813 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:04.703719 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:04.703747 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:04.707677 (+ 3930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:26:04.713657 (+ 5980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:04.713667 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:04.713672 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:04.714480 (+ 808us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:04.714856 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:04.714859 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:04.714862 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:04.715107 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:04.715797 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:26:04.715803 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:26:04.715989 (+ 186us) client_negotiation.cc:769] Sending connection context
0811 22:26:04.716169 (+ 180us) client_negotiation.cc:240] Negotiation successful
0811 22:26:04.716430 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1354,"thread_start_us":1016,"threads_started":1}
I20260811 22:26:04.716988 10901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:04.692653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42346 (local address 127.29.106.62:46573)
0811 22:26:04.693527 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:04.693532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:04.695113 (+ 1581us) server_negotiation.cc:407] Connection header received
0811 22:26:04.695380 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:04.695385 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:04.695524 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:04.695619 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:04.698085 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:04.699198 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:04.701492 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:04.702286 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:04.702407 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:04.708042 (+ 5635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:04.708053 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:04.708058 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:04.708105 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:04.713538 (+ 5433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:04.714594 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:04.714598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:04.714600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:04.714756 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:04.715213 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:04.715217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:04.715219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:04.715633 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:04.715744 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:04.716316 (+ 572us) server_negotiation.cc:299] Negotiation successful
0811 22:26:04.716488 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":733,"thread_start_us":592,"threads_started":1}
I20260811 22:26:04.718326 10892 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.106.61" port: 46755 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 22:26:04 dist-test-slave-gjrz krb5kdc[9387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487164, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.106.60@KRBTEST.COM for oryx/127.29.106.61@KRBTEST.COM
I20260811 22:26:04.736608 10902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:04.719412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.61:46755 (local address 127.0.0.1:56762)
0811 22:26:04.719625 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:04.719644 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:04.719742 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:04.720039 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:04.720043 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:04.720468 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:04.720683 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:04.720693 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:04.721940 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:04.721945 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:04.723357 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:04.723370 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:04.724373 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:04.724378 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:04.724714 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:04.725660 (+ 946us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:04.725677 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:04.729430 (+ 3753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:26:04.733856 (+ 4426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:04.733859 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:04.733862 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:04.734468 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:04.734871 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:04.734874 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:04.734876 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:04.735126 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:04.735702 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:26:04.735706 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:26:04.735868 (+ 162us) client_negotiation.cc:769] Sending connection context
0811 22:26:04.735980 (+ 112us) client_negotiation.cc:240] Negotiation successful
0811 22:26:04.736277 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260811 22:26:04.736805 10756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:04.719288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56762 (local address 127.29.106.61:46755)
0811 22:26:04.719466 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:04.719470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:04.719698 (+ 228us) server_negotiation.cc:407] Connection header received
0811 22:26:04.719837 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:04.719841 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:04.719909 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:04.719983 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:04.720811 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:04.721777 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:04.723484 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:04.724195 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:04.724291 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:04.729652 (+ 5361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:04.729658 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:04.729661 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:04.729694 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:04.733735 (+ 4041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:04.734565 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:04.734569 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:04.734571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:04.734784 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:04.735217 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:04.735220 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:04.735222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:04.735605 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:04.735718 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:04.736107 (+ 389us) server_negotiation.cc:299] Negotiation successful
0811 22:26:04.736334 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260811 22:26:04.737884 10892 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.106.60" port: 40553 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 22:26:04 dist-test-slave-gjrz krb5kdc[9387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487164, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.106.60@KRBTEST.COM for oryx/127.29.106.60@KRBTEST.COM
I20260811 22:26:04.755219 10902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:04.739086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:40553 (local address 127.0.0.1:47096)
0811 22:26:04.739301 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:04.739317 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:04.739413 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:04.739668 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:04.739672 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:04.740206 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:04.740480 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:04.740490 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:04.741668 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:04.741674 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:04.742843 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:04.742856 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:04.743896 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:04.743902 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:04.744224 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:04.744887 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:04.744903 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:04.747654 (+ 2751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:26:04.751851 (+ 4197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:04.751857 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:04.751861 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:04.752414 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:04.752912 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:04.752917 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:04.752920 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:04.753215 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:04.753818 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:26:04.753823 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:26:04.754093 (+ 270us) client_negotiation.cc:769] Sending connection context
0811 22:26:04.754252 (+ 159us) client_negotiation.cc:240] Negotiation successful
0811 22:26:04.754523 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260811 22:26:04.755539 10899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:04.738652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47096 (local address 127.29.106.60:40553)
0811 22:26:04.738829 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:04.738839 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:04.739380 (+ 541us) server_negotiation.cc:407] Connection header received
0811 22:26:04.739504 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:04.739508 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:04.739563 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:04.739637 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:04.740613 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:04.741543 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:04.742972 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:04.743732 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:04.743842 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:04.747861 (+ 4019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:04.747865 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:04.747869 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:04.747898 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:04.751694 (+ 3796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:04.752525 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:04.752529 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:04.752531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:04.752766 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:04.753323 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:04.753326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:04.753328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:04.753704 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:04.753837 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:04.754401 (+ 564us) server_negotiation.cc:299] Negotiation successful
0811 22:26:04.754786 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":58}
I20260811 22:26:04.761977 10892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79: Bootstrap starting.
I20260811 22:26:04.767159 10892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79: Neither blocks nor log segments found. Creating new log.
I20260811 22:26:04.768878 10892 log.cc:824] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79: Log is configured to *not* fsync() on all Append() calls
I20260811 22:26:04.772181 10892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79: No bootstrap required, opened a new log
I20260811 22:26:04.784372 10892 raft_consensus.cc:348] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 46755 } } peers { permanent_uuid: "800b0066b8d34ef98b9c2bc24409ee79" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 40553 } }
I20260811 22:26:04.784818 10892 raft_consensus.cc:374] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:26:04.785029 10892 raft_consensus.cc:729] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 800b0066b8d34ef98b9c2bc24409ee79, State: Initialized, Role: FOLLOWER
I20260811 22:26:04.785691 10892 consensus_queue.cc:260] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 46755 } } peers { permanent_uuid: "800b0066b8d34ef98b9c2bc24409ee79" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 40553 } }
I20260811 22:26:04.787379 10904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 46755 } } peers { permanent_uuid: "800b0066b8d34ef98b9c2bc24409ee79" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 40553 } } }
I20260811 22:26:04.788000 10904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:26:04.788165 10892 sys_catalog.cc:564] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [sys.catalog]: configured and running, proceeding with master startup.
W20260811 22:26:04.806416 10921 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 22:26:04.806704 10921 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 22:26:04.810536 10567 master_runner.cc:428] Detected that this master 800b0066b8d34ef98b9c2bc24409ee79 is joining an existing cluster
I20260811 22:26:04.810847 10567 master_runner.cc:432] Initiating AddMaster RPC to add 127.29.106.60:40553
I20260811 22:26:04.814802 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 9599
I20260811 22:26:04.829730 10904 raft_consensus.cc:482] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:26:04.830268 10904 raft_consensus.cc:504] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 46755 } } peers { permanent_uuid: "800b0066b8d34ef98b9c2bc24409ee79" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 40553 } }
I20260811 22:26:04.832561 10904 leader_election.cc:288] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b5d1ed0b161417688f43a878b41ef3d (127.29.106.62:46573), e0ad6642bf744796b25dfefd04d3f245 (127.29.106.61:46755)
I20260811 22:26:04.834067 9503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "800b0066b8d34ef98b9c2bc24409ee79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b5d1ed0b161417688f43a878b41ef3d" is_pre_election: true
I20260811 22:26:04.834846 10669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "800b0066b8d34ef98b9c2bc24409ee79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0ad6642bf744796b25dfefd04d3f245" is_pre_election: true
I20260811 22:26:04.836920 10786 leader_election.cc:302] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 800b0066b8d34ef98b9c2bc24409ee79; no voters: 2b5d1ed0b161417688f43a878b41ef3d, e0ad6642bf744796b25dfefd04d3f245
Aug 11 22:26:04 dist-test-slave-gjrz krb5kdc[9387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.106.1@KRBTEST.COM
I20260811 22:26:04.841212 10904 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260811 22:26:04.855294 10931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:04.816635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40046 (local address 127.29.106.1:35211)
0811 22:26:04.817835 (+ 1200us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:04.817843 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:04.817867 (+ 24us) server_negotiation.cc:407] Connection header received
0811 22:26:04.817923 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:04.817930 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:04.818150 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:04.818311 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:04.821503 (+ 3192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:04.823884 (+ 2381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:04.826386 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:04.827633 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:04.842875 (+ 15242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:04.843382 (+ 507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:04.843400 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:04.843415 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:04.843526 (+ 111us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:04.849433 (+ 5907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:04.850586 (+ 1153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:04.850596 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:04.850609 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:04.850896 (+ 287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:04.851504 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:04.851509 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:04.851513 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:04.852253 (+ 740us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:04.854142 (+ 1889us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:04.854307 (+ 165us) server_negotiation.cc:299] Negotiation successful
0811 22:26:04.854621 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":991,"thread_start_us":752,"threads_started":1}
I20260811 22:26:04.857016 9842 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:26:04.867080 10756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:04.819381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56766 (local address 127.29.106.61:46755)
0811 22:26:04.819728 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:04.819731 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:04.838392 (+ 18661us) server_negotiation.cc:407] Connection header received
0811 22:26:04.839304 (+ 912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:04.839311 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:04.839414 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:04.839490 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:04.844542 (+ 5052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:04.845911 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:04.849946 (+ 4035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:04.850859 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:04.850975 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:04.854963 (+ 3988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:04.854969 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:04.854973 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:04.855014 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:04.859779 (+ 4765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:04.860815 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:04.860821 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:04.860824 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:04.860987 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:04.863626 (+ 2639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:04.863632 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:04.863635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:04.864127 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:04.864261 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:04.866323 (+ 2062us) server_negotiation.cc:299] Negotiation successful
0811 22:26:04.866694 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231}
W20260811 22:26:04.865955 10794 tablet.cc:2369] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:26:04.870018 10937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:04.821165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.61:46755 (local address 127.0.0.1:56766)
0811 22:26:04.838224 (+ 17059us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:04.838256 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:04.838751 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:04.839787 (+ 1036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:04.839793 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:04.842185 (+ 2392us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:04.844340 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:04.844355 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:04.846142 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:04.846147 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:04.849767 (+ 3620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:04.849782 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:04.851376 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:04.851384 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:04.851583 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:04.852683 (+ 1100us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:04.852708 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:04.854713 (+ 2005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:26:04.860022 (+ 5309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:04.860027 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:04.860031 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:04.860674 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:04.861123 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:04.861126 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:04.861128 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:04.863449 (+ 2321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:04.864236 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:26:04.864241 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:26:04.865706 (+ 1465us) client_negotiation.cc:769] Sending connection context
0811 22:26:04.866133 (+ 427us) client_negotiation.cc:240] Negotiation successful
0811 22:26:04.868752 (+ 2619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15293,"thread_start_us":22016,"threads_started":1}
I20260811 22:26:04.883204 10933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:04.821312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:40553 (local address 127.0.0.1:47104)
0811 22:26:04.843411 (+ 22099us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:04.843432 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:04.843647 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:04.843957 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:04.843962 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:04.846210 (+ 2248us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:04.848446 (+ 2236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:04.848462 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:04.852042 (+ 3580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:04.852047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:04.862097 (+ 10050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:04.862112 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:04.865353 (+ 3241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:04.865363 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:04.867611 (+ 2248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:04.869243 (+ 1632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:04.869267 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:04.873347 (+ 4080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:26:04.879402 (+ 6055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:04.879407 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:04.879411 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:04.880107 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:04.880613 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:04.880619 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:04.880623 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:04.880835 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:04.881569 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:26:04.881573 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:26:04.881755 (+ 182us) client_negotiation.cc:769] Sending connection context
0811 22:26:04.881916 (+ 161us) client_negotiation.cc:240] Negotiation successful
0811 22:26:04.882224 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19824,"thread_start_us":17525,"threads_started":1}
I20260811 22:26:04.884258 10899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:04.820309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47104 (local address 127.29.106.60:40553)
0811 22:26:04.820546 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:04.820561 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:04.843516 (+ 22955us) server_negotiation.cc:407] Connection header received
0811 22:26:04.843776 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:04.843782 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:04.843841 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:04.843911 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:04.848646 (+ 4735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:04.850389 (+ 1743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:04.862242 (+ 11853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:04.864605 (+ 2363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:04.864723 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:04.873549 (+ 8826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:04.873555 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:04.873560 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:04.873595 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:04.879208 (+ 5613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:04.880236 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:04.880242 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:04.880245 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:04.880490 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:04.880947 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:04.880966 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:04.880969 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:04.881354 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:04.881517 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:04.883427 (+ 1910us) server_negotiation.cc:299] Negotiation successful
0811 22:26:04.883630 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260811 22:26:04.893333 10901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:04.819255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42360 (local address 127.29.106.62:46573)
0811 22:26:04.819584 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:04.819590 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:04.844820 (+ 25230us) server_negotiation.cc:407] Connection header received
0811 22:26:04.846958 (+ 2138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:04.846966 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:04.847054 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:04.847117 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:04.852425 (+ 5308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:04.853746 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:04.858173 (+ 4427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:04.859026 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:04.859149 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:04.885835 (+ 26686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:04.885843 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:04.885848 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:04.885894 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:04.889343 (+ 3449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:04.890426 (+ 1083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:04.890434 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:04.890437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:04.890686 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:04.891220 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:04.891225 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:04.891229 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:04.891676 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:04.891817 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:04.892587 (+ 770us) server_negotiation.cc:299] Negotiation successful
0811 22:26:04.892853 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167}
I20260811 22:26:04.894492 10932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:04.827951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:46573 (local address 127.0.0.1:42360)
0811 22:26:04.844705 (+ 16754us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:04.844724 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:04.846729 (+ 2005us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:04.847355 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:04.847359 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:04.851224 (+ 3865us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:04.852278 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:04.852291 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:04.854050 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:04.854055 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:04.857988 (+ 3933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:04.858003 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:04.859592 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:04.859602 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:04.859892 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:04.867579 (+ 7687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:04.867600 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:04.885563 (+ 17963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:26:04.889513 (+ 3950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:04.889519 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:04.889523 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:04.890273 (+ 750us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:04.890844 (+ 571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:04.890849 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:04.890852 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:04.891075 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:04.891907 (+ 832us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:26:04.891911 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:26:04.892205 (+ 294us) client_negotiation.cc:769] Sending connection context
0811 22:26:04.893516 (+ 1311us) client_negotiation.cc:240] Negotiation successful
0811 22:26:04.893891 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16559,"thread_start_us":9004,"threads_started":1}
Aug 11 22:26:04 dist-test-slave-gjrz krb5kdc[9387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487161, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.106.62@KRBTEST.COM for oryx/127.29.106.60@KRBTEST.COM
I20260811 22:26:04.917358 10755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:04.901035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:40553 (local address 127.0.0.1:47120)
0811 22:26:04.901301 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:04.901317 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:04.901421 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:04.901806 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:04.901811 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:04.902272 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:04.902460 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:04.902475 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:04.903826 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:04.903832 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:04.904855 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:04.904868 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:04.905732 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:04.905740 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:04.905968 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:04.906535 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:04.906553 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:04.909087 (+ 2534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:26:04.914246 (+ 5159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:04.914252 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:04.914257 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:04.914822 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:04.915238 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:04.915244 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:04.915248 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:04.915456 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:04.916179 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:26:04.916184 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:26:04.916391 (+ 207us) client_negotiation.cc:769] Sending connection context
0811 22:26:04.916528 (+ 137us) client_negotiation.cc:240] Negotiation successful
0811 22:26:04.916834 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162}
I20260811 22:26:04.917759 10899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:04.901061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47120 (local address 127.29.106.60:40553)
0811 22:26:04.901217 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:04.901222 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:04.901384 (+ 162us) server_negotiation.cc:407] Connection header received
0811 22:26:04.901567 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:04.901573 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:04.901673 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:04.901753 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:04.902645 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:04.903675 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:04.905001 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:04.905550 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:04.905672 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:04.909361 (+ 3689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:04.909367 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:04.909377 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:04.909409 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:04.914106 (+ 4697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:04.914945 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:04.914950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:04.914952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:04.915115 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:04.915567 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:04.915571 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:04.915573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:04.916060 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:04.916193 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:04.916647 (+ 454us) server_negotiation.cc:299] Negotiation successful
0811 22:26:04.916905 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 22:26:04.919644 9487 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.29.106.60:40553: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "800b0066b8d34ef98b9c2bc24409ee79"
member_type: NON_VOTER
last_known_addr {
host: "127.29.106.60"
port: 40553
}
attrs {
promote: true
}
}
dest_uuid: "2b5d1ed0b161417688f43a878b41ef3d"
cas_config_opid_index: 5
I20260811 22:26:04.922297 9487 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: NON_VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: true } } peers { permanent_uuid: "800b0066b8d34ef98b9c2bc24409ee79" member_type: NON_VOTER last_known_addr { host: "127.29.106.60" port: 40553 } attrs { promote: true } }
I20260811 22:26:04.943167 10751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2b5d1ed0b161417688f43a878b41ef3d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: NON_VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: true } } } pending_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: NON_VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: true } } peers { permanent_uuid: "800b0066b8d34ef98b9c2bc24409ee79" member_type: NON_VOTER last_known_addr { host: "127.29.106.60" port: 40553 } attrs { promote: true } } }
I20260811 22:26:04.943810 10751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d [sys.catalog]: This master's current role is: LEADER
I20260811 22:26:04.944146 10940 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 800b0066b8d34ef98b9c2bc24409ee79 (127.29.106.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: NON_VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: true } } peers { permanent_uuid: "800b0066b8d34ef98b9c2bc24409ee79" member_type: NON_VOTER last_known_addr { host: "127.29.106.60" port: 40553 } attrs { promote: true } } }
I20260811 22:26:04.954025 10843 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:26:04.955744 10669 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245 [term 1 LEARNER]: Refusing update from remote peer 2b5d1ed0b161417688f43a878b41ef3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260811 22:26:04.956748 10751 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: NON_VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260811 22:26:04.960017 10843 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [term 1 FOLLOWER]: Refusing update from remote peer 2b5d1ed0b161417688f43a878b41ef3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260811 22:26:04.959823 10669 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 800b0066b8d34ef98b9c2bc24409ee79 (127.29.106.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: NON_VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: true } } peers { permanent_uuid: "800b0066b8d34ef98b9c2bc24409ee79" member_type: NON_VOTER last_known_addr { host: "127.29.106.60" port: 40553 } attrs { promote: true } } }
I20260811 22:26:04.960881 10751 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "800b0066b8d34ef98b9c2bc24409ee79" member_type: NON_VOTER last_known_addr { host: "127.29.106.60" port: 40553 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:26:04.967597 10940 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.29.106.60:40553
I20260811 22:26:04.968714 10567 master.cc:537] Master@127.29.106.60:40553 shutting down...
I20260811 22:26:04.968648 10939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2b5d1ed0b161417688f43a878b41ef3d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: NON_VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: true } } peers { permanent_uuid: "800b0066b8d34ef98b9c2bc24409ee79" member_type: NON_VOTER last_known_addr { host: "127.29.106.60" port: 40553 } attrs { promote: true } } }
I20260811 22:26:04.969614 10939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d [sys.catalog]: This master's current role is: LEADER
I20260811 22:26:04.970605 10949 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d: attempting to promote NON_VOTER e0ad6642bf744796b25dfefd04d3f245 to VOTER
I20260811 22:26:04.970045 10952 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2b5d1ed0b161417688f43a878b41ef3d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: NON_VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: true } } peers { permanent_uuid: "800b0066b8d34ef98b9c2bc24409ee79" member_type: NON_VOTER last_known_addr { host: "127.29.106.60" port: 40553 } attrs { promote: true } } }
I20260811 22:26:04.970963 10952 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245 [sys.catalog]: This master's current role is: LEARNER
I20260811 22:26:04.971567 10949 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: false } } peers { permanent_uuid: "800b0066b8d34ef98b9c2bc24409ee79" member_type: NON_VOTER last_known_addr { host: "127.29.106.60" port: 40553 } attrs { promote: true } }
I20260811 22:26:04.976032 10669 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245 [term 1 LEARNER]: Refusing update from remote peer 2b5d1ed0b161417688f43a878b41ef3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260811 22:26:04.977030 10940 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260811 22:26:04.979976 10949 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e0ad6642bf744796b25dfefd04d3f245 (127.29.106.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: false } } peers { permanent_uuid: "800b0066b8d34ef98b9c2bc24409ee79" member_type: NON_VOTER last_known_addr { host: "127.29.106.60" port: 40553 } attrs { promote: true } } }
I20260811 22:26:04.980887 10669 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e0ad6642bf744796b25dfefd04d3f245 (127.29.106.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: false } } peers { permanent_uuid: "800b0066b8d34ef98b9c2bc24409ee79" member_type: NON_VOTER last_known_addr { host: "127.29.106.60" port: 40553 } attrs { promote: true } } }
I20260811 22:26:04.984418 10940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2b5d1ed0b161417688f43a878b41ef3d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: false } } peers { permanent_uuid: "800b0066b8d34ef98b9c2bc24409ee79" member_type: NON_VOTER last_known_addr { host: "127.29.106.60" port: 40553 } attrs { promote: true } } }
I20260811 22:26:04.985040 10940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d [sys.catalog]: This master's current role is: LEADER
I20260811 22:26:04.985430 10939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2b5d1ed0b161417688f43a878b41ef3d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: false } } peers { permanent_uuid: "800b0066b8d34ef98b9c2bc24409ee79" member_type: NON_VOTER last_known_addr { host: "127.29.106.60" port: 40553 } attrs { promote: true } } }
I20260811 22:26:04.985698 10939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d [sys.catalog]: This master's current role is: LEADER
I20260811 22:26:04.987648 10952 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2b5d1ed0b161417688f43a878b41ef3d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: false } } peers { permanent_uuid: "800b0066b8d34ef98b9c2bc24409ee79" member_type: NON_VOTER last_known_addr { host: "127.29.106.60" port: 40553 } attrs { promote: true } } }
I20260811 22:26:04.988039 10952 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245 [sys.catalog]: This master's current role is: FOLLOWER
W20260811 22:26:04.988550 9453 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d -> Peer 800b0066b8d34ef98b9c2bc24409ee79 (127.29.106.60:40553): Couldn't send request to peer 800b0066b8d34ef98b9c2bc24409ee79. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260811 22:26:04.989759 10843 raft_consensus.cc:858] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 46755 } } peers { permanent_uuid: "800b0066b8d34ef98b9c2bc24409ee79" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 40553 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: NON_VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: NON_VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: true } } peers { permanent_uuid: "800b0066b8d34ef98b9c2bc24409ee79" member_type: NON_VOTER last_known_addr { host: "127.29.106.60" port: 40553 } attrs { promote: true } }
I20260811 22:26:04.998148 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 9599
W20260811 22:26:04.990202 10843 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 46755 } } peers { permanent_uuid: "800b0066b8d34ef98b9c2bc24409ee79" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 40553 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: NON_VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: NON_VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: true } } peers { permanent_uuid: "800b0066b8d34ef98b9c2bc24409ee79" member_type: NON_VOTER last_known_addr { host: "127.29.106.60" port: 40553 } attrs { promote: true } }
I20260811 22:26:05.029603 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.1:35211
--local_ip_for_outbound_sockets=127.29.106.1
--tserver_master_addrs=127.29.106.62:46573
--webserver_port=38993
--webserver_interface=127.29.106.1
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.1.keytab
--principal=oryx/127.29.106.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:34763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.29.106.62:46573,127.29.106.61:46755
--tserver_master_addrs=127.29.106.62:46573,127.29.106.61:46755,127.29.106.60:40553 with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 22:26:05.039685 10843 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER e0ad6642bf744796b25dfefd04d3f245 (127.29.106.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: NON_VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: true } } }
I20260811 22:26:05.044837 10959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2b5d1ed0b161417688f43a878b41ef3d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: NON_VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: true } } }
I20260811 22:26:05.045619 10959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260811 22:26:05.046072 10904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b5d1ed0b161417688f43a878b41ef3d. Latest consensus state: current_term: 1 leader_uuid: "2b5d1ed0b161417688f43a878b41ef3d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: NON_VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: true } } }
I20260811 22:26:05.046427 10904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260811 22:26:05.046763 10965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2b5d1ed0b161417688f43a878b41ef3d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: NON_VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: true } } }
I20260811 22:26:05.047454 10965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260811 22:26:05.048352 10567 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260811 22:26:05.048810 10567 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260811 22:26:05.048995 10567 tablet_replica.cc:330] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79: stopping tablet replica
I20260811 22:26:05.056918 10567 master.cc:552] Master@127.29.106.60:40553 shutdown complete.
I20260811 22:26:05.057003 10567 master_runner.cc:305] Clearing existing system tablet
I20260811 22:26:05.057348 10567 fs_manager.cc:359] Metadata directory not provided
I20260811 22:26:05.057438 10567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:26:05.059597 10567 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.002s
I20260811 22:26:05.060976 10971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:05.061316 10567 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:26:05.061553 10567 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260811 22:26:05.061650 10567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-2/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-2/wal
uuid: "800b0066b8d34ef98b9c2bc24409ee79"
format_stamp: "Formatted at 2026-08-11 22:26:04 on dist-test-slave-gjrz"
server_key: "cd3165ef52498f1aae0982fec35e6881"
server_key_iv: "97480168abee36db1789e58a0ff7b6be"
server_key_version: "encryptionkey@0"
I20260811 22:26:05.067659 10567 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:26:05.076156 10567 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260811 22:26:05.076449 10567 log.cc:1196] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260811 22:26:05.077514 10567 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79: Deleting consensus metadata
I20260811 22:26:05.077873 10567 master_runner.cc:315] Copying system tablet from 127.29.106.62:46573
I20260811 22:26:05.112561 10567 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79: tablet copy: Beginning tablet copy session from remote peer at address 127.29.106.62:46573
I20260811 22:26:05.144626 10980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:05.113864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:46573 (local address 127.0.0.1:42362)
0811 22:26:05.127329 (+ 13465us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:05.127363 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:05.127556 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:05.128220 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:05.128226 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:05.128962 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:05.129246 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:05.129260 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:05.131296 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:05.131302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:05.133036 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:05.133052 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:05.134175 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:05.134181 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:05.134461 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:05.135226 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:05.135255 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:05.136620 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:26:05.141292 (+ 4672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:05.141298 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:05.141302 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:05.141830 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:05.142345 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:05.142350 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:05.142353 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:05.142596 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:05.143303 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:26:05.143307 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:26:05.143552 (+ 245us) client_negotiation.cc:769] Sending connection context
0811 22:26:05.143702 (+ 150us) client_negotiation.cc:240] Negotiation successful
0811 22:26:05.144055 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13274,"thread_start_us":8864,"threads_started":1}
I20260811 22:26:05.144626 10901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:05.116025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42362 (local address 127.29.106.62:46573)
0811 22:26:05.116262 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:05.116269 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:05.127487 (+ 11218us) server_negotiation.cc:407] Connection header received
0811 22:26:05.127820 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:05.127828 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:05.127926 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:05.128047 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:05.129416 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:05.131089 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:05.133186 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:05.134027 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:05.134161 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:05.136842 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:05.136851 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:05.136856 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:05.136898 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:05.141171 (+ 4273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:05.142025 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:05.142031 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:05.142035 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:05.142233 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:05.142725 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:05.142729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:05.142733 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:05.143158 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:05.143371 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:05.143908 (+ 537us) server_negotiation.cc:299] Negotiation successful
0811 22:26:05.144144 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260811 22:26:05.145354 9520 tablet_copy_service.cc:140] P 2b5d1ed0b161417688f43a878b41ef3d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 800b0066b8d34ef98b9c2bc24409ee79 ({username='oryx', principal='oryx/127.29.106.60@KRBTEST.COM'} at 127.0.0.1:42362)
I20260811 22:26:05.145597 9520 tablet_copy_service.cc:161] P 2b5d1ed0b161417688f43a878b41ef3d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 800b0066b8d34ef98b9c2bc24409ee79 at {username='oryx', principal='oryx/127.29.106.60@KRBTEST.COM'} at 127.0.0.1:42362: session id = 800b0066b8d34ef98b9c2bc24409ee79-00000000000000000000000000000000
I20260811 22:26:05.148984 9520 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d: Tablet Copy: opened 0 blocks and 1 log segments
I20260811 22:26:05.150727 10567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260811 22:26:05.156283 10567 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79: tablet copy: Starting download of 0 data blocks...
I20260811 22:26:05.156575 10567 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79: tablet copy: Starting download of 1 WAL segments...
I20260811 22:26:05.158811 10567 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260811 22:26:05.162196 9520 tablet_copy_service.cc:342] P 2b5d1ed0b161417688f43a878b41ef3d: Request end of tablet copy session 800b0066b8d34ef98b9c2bc24409ee79-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.29.106.60@KRBTEST.COM'} at 127.0.0.1:42362
I20260811 22:26:05.162357 9520 tablet_copy_service.cc:433] P 2b5d1ed0b161417688f43a878b41ef3d: ending tablet copy session 800b0066b8d34ef98b9c2bc24409ee79-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 800b0066b8d34ef98b9c2bc24409ee79
I20260811 22:26:05.172392 10567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:26:05.174954 10981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:05.176554 10982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:26:05.197050 10567 server_base.cc:1034] running on GCE node
W20260811 22:26:05.197050 10988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:26:05.197626 10567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:26:05.198287 10567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:26:05.199467 10567 hybrid_clock.cc:648] HybridClock initialized: now 1786487165199394 us; error 58 us; skew 500 ppm
I20260811 22:26:05.199683 10567 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:26:05 dist-test-slave-gjrz krb5kdc[9387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487165, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.106.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:26:05.201655 10567 init.cc:377] Logged in from keytab as oryx/127.29.106.60@KRBTEST.COM (short username oryx)
I20260811 22:26:05.203837 10567 webserver.cc:460] Webserver started at http://127.29.106.60:42543/ using document root <none> and password file <none>
I20260811 22:26:05.204138 10567 fs_manager.cc:359] Metadata directory not provided
I20260811 22:26:05.204229 10567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:26:05.206547 10567 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260811 22:26:05.208487 11003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:05.208885 10567 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 22:26:05.209021 10567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-2/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-2/wal
uuid: "800b0066b8d34ef98b9c2bc24409ee79"
format_stamp: "Formatted at 2026-08-11 22:26:04 on dist-test-slave-gjrz"
server_key: "cd3165ef52498f1aae0982fec35e6881"
server_key_iv: "97480168abee36db1789e58a0ff7b6be"
server_key_version: "encryptionkey@0"
I20260811 22:26:05.209263 10567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:26:05.247727 10567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:26:05.251078 10567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:26:05.277801 10567 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.60:40553
I20260811 22:26:05.277837 11104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.60:40553 every 8 connection(s)
I20260811 22:26:05.278735 10567 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/master-2/data/info.pb
I20260811 22:26:05.285046 11106 sys_catalog.cc:263] Verifying existing consensus state
I20260811 22:26:05.288672 11106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79: Bootstrap starting.
I20260811 22:26:05.300571 11106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 22:26:05.300990 11106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79: Bootstrap complete.
I20260811 22:26:05.301785 11106 raft_consensus.cc:348] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: false } } peers { permanent_uuid: "800b0066b8d34ef98b9c2bc24409ee79" member_type: NON_VOTER last_known_addr { host: "127.29.106.60" port: 40553 } attrs { promote: true } }
I20260811 22:26:05.302038 11106 raft_consensus.cc:729] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 800b0066b8d34ef98b9c2bc24409ee79, State: Initialized, Role: LEARNER
I20260811 22:26:05.302270 11106 consensus_queue.cc:260] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: false } } peers { permanent_uuid: "800b0066b8d34ef98b9c2bc24409ee79" member_type: NON_VOTER last_known_addr { host: "127.29.106.60" port: 40553 } attrs { promote: true } }
I20260811 22:26:05.311856 11112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: false } } peers { permanent_uuid: "800b0066b8d34ef98b9c2bc24409ee79" member_type: NON_VOTER last_known_addr { host: "127.29.106.60" port: 40553 } attrs { promote: true } } }
I20260811 22:26:05.312237 11106 sys_catalog.cc:564] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:26:05.312414 11112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [sys.catalog]: This master's current role is: LEARNER
I20260811 22:26:05.320308 11134 catalog_manager.cc:1259] Loaded cluster ID: 8731d947535c4e8c8f3224f1af27f098
I20260811 22:26:05.320394 11134 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79: loading cluster ID for follower catalog manager: success
I20260811 22:26:05.322959 11134 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79: acquiring CA information for follower catalog manager: success
I20260811 22:26:05.324772 11134 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 22:26:05.357134 10755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:05.350536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:40553 (local address 127.0.0.1:47128)
0811 22:26:05.350838 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:05.350860 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:05.351020 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:05.351920 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:05.351925 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:05.351957 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:26:05.352129 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:05.352138 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:05.353454 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:05.353459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:05.354897 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:05.354918 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:05.356279 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:05.356286 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:05.356508 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:05.356553 (+ 45us) client_negotiation.cc:769] Sending connection context
0811 22:26:05.356649 (+ 96us) client_negotiation.cc:240] Negotiation successful
0811 22:26:05.356796 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160}
I20260811 22:26:05.357136 11136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:05.350695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47128 (local address 127.29.106.60:40553)
0811 22:26:05.351568 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:05.351572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:05.351589 (+ 17us) server_negotiation.cc:407] Connection header received
0811 22:26:05.351659 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:05.351662 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:05.351748 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:05.351854 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:26:05.352286 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:05.353295 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:05.355088 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:05.356080 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:05.356262 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:05.356550 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:05.356684 (+ 134us) server_negotiation.cc:299] Negotiation successful
0811 22:26:05.356813 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":753,"thread_start_us":559,"threads_started":1}
I20260811 22:26:05.358012 11061 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260811 22:26:05.358428 11112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b5d1ed0b161417688f43a878b41ef3d. Latest consensus state: current_term: 1 leader_uuid: "2b5d1ed0b161417688f43a878b41ef3d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: false } } peers { permanent_uuid: "800b0066b8d34ef98b9c2bc24409ee79" member_type: NON_VOTER last_known_addr { host: "127.29.106.60" port: 40553 } attrs { promote: true } } }
I20260811 22:26:05.358742 11112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [sys.catalog]: This master's current role is: LEARNER
W20260811 22:26:05.564357 10963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:26:05.564633 10963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:26:05.564743 10963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:26:05.574121 10963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:26:05.574258 10963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:26:05.574433 10963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.1
I20260811 22:26:05.593680 10963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:34763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.1.keytab
--principal=oryx/127.29.106.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.1:35211
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.106.1
--webserver_port=38993
--tserver_master_addrs=127.29.106.62:46573,127.29.106.61:46755,127.29.106.60:40553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.1
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:26:05.595325 10963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:26:05.597141 10963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:26:05.615666 11142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:05.615906 11143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:05.616030 11145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:26:05.617456 10963 server_base.cc:1034] running on GCE node
I20260811 22:26:05.618263 10963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:26:05.619458 10963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:26:05.620759 10963 hybrid_clock.cc:648] HybridClock initialized: now 1786487165620635 us; error 44 us; skew 500 ppm
I20260811 22:26:05.621264 10963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:26:05 dist-test-slave-gjrz krb5kdc[9387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487165, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.106.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:26:05.626713 10963 init.cc:377] Logged in from keytab as oryx/127.29.106.1@KRBTEST.COM (short username oryx)
I20260811 22:26:05.628892 10963 webserver.cc:460] Webserver started at http://127.29.106.1:38993/ using document root <none> and password file <none>
I20260811 22:26:05.629843 10963 fs_manager.cc:359] Metadata directory not provided
I20260811 22:26:05.630092 10963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:26:05.637331 10963 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 22:26:05.641366 11156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:05.642982 10963 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260811 22:26:05.643220 10963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-0/wal
uuid: "f16cafe5e19446e782da1ed0fcd4a964"
format_stamp: "Formatted at 2026-08-11 22:26:02 on dist-test-slave-gjrz"
server_key: "59c3ff62a5f6d611f812249a934eea37"
server_key_iv: "d735da4e44ebe03d0937e108dfe182ce"
server_key_version: "encryptionkey@0"
I20260811 22:26:05.644896 10963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:26:05.664876 10963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:26:05.670998 10963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:26:05.671568 10963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:26:05.673516 10963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:26:05.676122 10963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:26:05.676376 10963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:05.676544 10963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:26:05.676787 10963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:05.748332 10949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2b5d1ed0b161417688f43a878b41ef3d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: false } } peers { permanent_uuid: "800b0066b8d34ef98b9c2bc24409ee79" member_type: NON_VOTER last_known_addr { host: "127.29.106.60" port: 40553 } attrs { promote: true } } }
I20260811 22:26:05.748888 10949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d [sys.catalog]: This master's current role is: LEADER
I20260811 22:26:05.760604 11307 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d: attempting to promote NON_VOTER 800b0066b8d34ef98b9c2bc24409ee79 to VOTER
I20260811 22:26:05.761523 11307 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: false } } peers { permanent_uuid: "800b0066b8d34ef98b9c2bc24409ee79" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 40553 } attrs { promote: false } }
I20260811 22:26:05.763938 11061 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [term 1 LEARNER]: Refusing update from remote peer 2b5d1ed0b161417688f43a878b41ef3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260811 22:26:05.763931 10669 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245 [term 1 FOLLOWER]: Refusing update from remote peer 2b5d1ed0b161417688f43a878b41ef3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260811 22:26:05.764700 11307 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "800b0066b8d34ef98b9c2bc24409ee79" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 40553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260811 22:26:05.765251 11309 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260811 22:26:05.768714 11309 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 800b0066b8d34ef98b9c2bc24409ee79 (127.29.106.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: false } } peers { permanent_uuid: "800b0066b8d34ef98b9c2bc24409ee79" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 40553 } attrs { promote: false } } }
I20260811 22:26:05.769416 10669 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 800b0066b8d34ef98b9c2bc24409ee79 (127.29.106.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: false } } peers { permanent_uuid: "800b0066b8d34ef98b9c2bc24409ee79" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 40553 } attrs { promote: false } } }
I20260811 22:26:05.769923 11061 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 800b0066b8d34ef98b9c2bc24409ee79 (127.29.106.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: false } } peers { permanent_uuid: "800b0066b8d34ef98b9c2bc24409ee79" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 40553 } attrs { promote: false } } }
I20260811 22:26:05.775494 10949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2b5d1ed0b161417688f43a878b41ef3d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: false } } peers { permanent_uuid: "800b0066b8d34ef98b9c2bc24409ee79" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 40553 } attrs { promote: false } } }
I20260811 22:26:05.775902 10949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d [sys.catalog]: This master's current role is: LEADER
I20260811 22:26:05.776171 11112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2b5d1ed0b161417688f43a878b41ef3d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: false } } peers { permanent_uuid: "800b0066b8d34ef98b9c2bc24409ee79" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 40553 } attrs { promote: false } } }
I20260811 22:26:05.776310 10949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2b5d1ed0b161417688f43a878b41ef3d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: false } } peers { permanent_uuid: "800b0066b8d34ef98b9c2bc24409ee79" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 40553 } attrs { promote: false } } }
I20260811 22:26:05.776690 11112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:26:05.776808 10949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d [sys.catalog]: This master's current role is: LEADER
I20260811 22:26:05.777156 11309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2b5d1ed0b161417688f43a878b41ef3d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: false } } peers { permanent_uuid: "800b0066b8d34ef98b9c2bc24409ee79" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 40553 } attrs { promote: false } } }
I20260811 22:26:05.777483 11309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d [sys.catalog]: This master's current role is: LEADER
I20260811 22:26:05.779210 11332 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2b5d1ed0b161417688f43a878b41ef3d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: false } } peers { permanent_uuid: "800b0066b8d34ef98b9c2bc24409ee79" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 40553 } attrs { promote: false } } }
I20260811 22:26:05.779920 11332 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:26:05.797497 10963 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.1:35211
I20260811 22:26:05.797808 11370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.1:35211 every 8 connection(s)
I20260811 22:26:05.799466 10963 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
I20260811 22:26:05.804075 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 10963
I20260811 22:26:05.804558 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 9862
Aug 11 22:26:05 dist-test-slave-gjrz krb5kdc[9387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.106.2@KRBTEST.COM
Aug 11 22:26:05 dist-test-slave-gjrz krb5kdc[9387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487165, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.106.1@KRBTEST.COM for oryx/127.29.106.60@KRBTEST.COM
Aug 11 22:26:05 dist-test-slave-gjrz krb5kdc[9387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487165, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.106.1@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM
Aug 11 22:26:05 dist-test-slave-gjrz krb5kdc[9387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487165, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.106.1@KRBTEST.COM for oryx/127.29.106.61@KRBTEST.COM
I20260811 22:26:05.866042 11136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:05.806920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:44905 (local address 127.29.106.60:40553)
0811 22:26:05.807165 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:05.807171 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:05.828778 (+ 21607us) server_negotiation.cc:407] Connection header received
0811 22:26:05.837438 (+ 8660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:05.837444 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:05.837527 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:05.837592 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:05.841910 (+ 4318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:05.843468 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:05.847407 (+ 3939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:05.848435 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:05.848543 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:05.855961 (+ 7418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:05.855968 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:05.855972 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:05.856014 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:05.861166 (+ 5152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:05.862320 (+ 1154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:05.862325 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:05.862328 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:05.862699 (+ 371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:05.863443 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:05.863448 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:05.863451 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:05.863843 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:05.863986 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:05.865137 (+ 1151us) server_negotiation.cc:299] Negotiation successful
0811 22:26:05.865411 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260811 22:26:05.866124 11389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:05.824857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:35561 (local address 127.29.106.61:46755)
0811 22:26:05.826182 (+ 1325us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:05.826189 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:05.837142 (+ 10953us) server_negotiation.cc:407] Connection header received
0811 22:26:05.837307 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:05.837312 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:05.837458 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:05.837556 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:05.842103 (+ 4547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:05.843909 (+ 1806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:05.847405 (+ 3496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:05.848395 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:05.848498 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:05.856550 (+ 8052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:05.856556 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:05.856562 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:05.856604 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:05.862335 (+ 5731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:05.863452 (+ 1117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:05.863456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:05.863459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:05.863647 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:05.864212 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:05.864217 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:05.864220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:05.864637 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:05.864764 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:05.865361 (+ 597us) server_negotiation.cc:299] Negotiation successful
0811 22:26:05.865581 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1147,"thread_start_us":807,"threads_started":1}
I20260811 22:26:05.867684 11385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:05.816970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:46573 (local address 127.29.106.1:58017)
0811 22:26:05.831473 (+ 14503us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:05.831496 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:05.836137 (+ 4641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:05.839305 (+ 3168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:05.839310 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:05.840769 (+ 1459us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:05.842016 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:05.842031 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:05.844139 (+ 2108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:05.844144 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:05.847218 (+ 3074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:05.847233 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:05.849066 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:05.849076 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:05.849644 (+ 568us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:05.851126 (+ 1482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:05.851148 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:05.855677 (+ 4529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:26:05.861732 (+ 6055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:05.861738 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:05.861742 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:05.862368 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:05.862977 (+ 609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:05.862982 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:05.862986 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:05.863358 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:05.864311 (+ 953us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:26:05.864315 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:26:05.864613 (+ 298us) client_negotiation.cc:769] Sending connection context
0811 22:26:05.864988 (+ 375us) client_negotiation.cc:240] Negotiation successful
0811 22:26:05.865567 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14205,"mutex_wait_us":181,"spinlock_wait_cycles":1280,"thread_start_us":13649,"threads_started":1}
I20260811 22:26:05.867691 11383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:05.814060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:40553 (local address 127.29.106.1:44905)
0811 22:26:05.828533 (+ 14473us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:05.828611 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:05.837279 (+ 8668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:05.838088 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:05.838092 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:05.839966 (+ 1874us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:05.841585 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:05.841613 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:05.843678 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:05.843683 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:05.847216 (+ 3533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:05.847232 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:05.849250 (+ 2018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:05.849262 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:05.849626 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:05.850704 (+ 1078us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:05.850737 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:05.855652 (+ 4915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:26:05.861330 (+ 5678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:05.861338 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:05.861356 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:05.862174 (+ 818us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:05.862861 (+ 687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:05.862866 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:05.862870 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:05.863305 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:05.864086 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:26:05.864094 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:26:05.864607 (+ 513us) client_negotiation.cc:769] Sending connection context
0811 22:26:05.866258 (+ 1651us) client_negotiation.cc:240] Negotiation successful
0811 22:26:05.866620 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13916,"mutex_wait_us":178,"spinlock_wait_cycles":2432,"thread_start_us":14624,"threads_started":1}
I20260811 22:26:05.868418 11378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:05.804163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:58017 (local address 127.29.106.62:46573)
0811 22:26:05.838941 (+ 34778us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:05.838948 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:05.838970 (+ 22us) server_negotiation.cc:407] Connection header received
0811 22:26:05.839038 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:05.839043 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:05.839129 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:05.839383 (+ 254us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:05.842245 (+ 2862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:05.843987 (+ 1742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:05.847499 (+ 3512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:05.848588 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:05.850325 (+ 1737us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:05.856002 (+ 5677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:05.856010 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:05.856014 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:05.856056 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:05.861582 (+ 5526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:05.862581 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:05.862586 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:05.862589 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:05.862834 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:05.863502 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:05.863507 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:05.863510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:05.864044 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:05.867298 (+ 3254us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:05.867464 (+ 166us) server_negotiation.cc:299] Negotiation successful
0811 22:26:05.867754 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34636,"thread_start_us":35101,"threads_started":1}
I20260811 22:26:05.868610 11390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:05.825016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.61:46755 (local address 127.29.106.1:35561)
0811 22:26:05.837011 (+ 11995us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:05.837026 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:05.837184 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:05.838074 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:05.838087 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:05.839695 (+ 1608us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:05.841679 (+ 1984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:05.841697 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:05.844177 (+ 2480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:05.844182 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:05.847218 (+ 3036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:05.847233 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:05.849092 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:05.849100 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:05.849647 (+ 547us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:05.850938 (+ 1291us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:05.851019 (+ 81us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:05.856278 (+ 5259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:26:05.862497 (+ 6219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:05.862501 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:05.862505 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:05.863321 (+ 816us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:05.863797 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:05.863802 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:05.863805 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:05.864088 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:05.864846 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:26:05.864851 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:26:05.865038 (+ 187us) client_negotiation.cc:769] Sending connection context
0811 22:26:05.866380 (+ 1342us) client_negotiation.cc:240] Negotiation successful
0811 22:26:05.866758 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11826,"mutex_wait_us":543,"spinlock_wait_cycles":137728,"thread_start_us":11437,"threads_started":1}
I20260811 22:26:05.870283 11373 heartbeater.cc:347] Connected to a master server at 127.29.106.62:46573
I20260811 22:26:05.870507 11374 heartbeater.cc:347] Connected to a master server at 127.29.106.61:46755
I20260811 22:26:05.870786 11374 heartbeater.cc:464] Registering TS with master...
I20260811 22:26:05.871002 11371 heartbeater.cc:347] Connected to a master server at 127.29.106.60:40553
I20260811 22:26:05.871197 11382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:05.806220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53990 (local address 127.29.106.2:39773)
0811 22:26:05.833934 (+ 27714us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:05.833944 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:05.833974 (+ 30us) server_negotiation.cc:407] Connection header received
0811 22:26:05.834066 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:05.834073 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:05.834300 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:05.834532 (+ 232us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:05.838486 (+ 3954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:05.840789 (+ 2303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:05.843775 (+ 2986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:05.845090 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:05.845220 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:05.856943 (+ 11723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:05.856961 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:05.856979 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:05.857023 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:05.863209 (+ 6186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:05.864571 (+ 1362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:05.864580 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:05.864583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:05.864869 (+ 286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:05.865391 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:05.865396 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:05.865399 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:05.865898 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:05.867008 (+ 1110us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:05.870130 (+ 3122us) server_negotiation.cc:299] Negotiation successful
0811 22:26:05.870470 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27549,"thread_start_us":27160,"threads_started":1}
I20260811 22:26:05.871227 11371 heartbeater.cc:464] Registering TS with master...
I20260811 22:26:05.870805 11373 heartbeater.cc:464] Registering TS with master...
I20260811 22:26:05.872790 11374 heartbeater.cc:511] Master 127.29.106.61:46755 requested a full tablet report, sending...
I20260811 22:26:05.873270 11371 heartbeater.cc:511] Master 127.29.106.60:40553 requested a full tablet report, sending...
I20260811 22:26:05.873286 10105 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:26:05.875882 10653 ts_manager.cc:194] Registered new tserver with Master: f16cafe5e19446e782da1ed0fcd4a964 (127.29.106.1:35211)
I20260811 22:26:05.876001 11046 ts_manager.cc:194] Registered new tserver with Master: f16cafe5e19446e782da1ed0fcd4a964 (127.29.106.1:35211)
I20260811 22:26:05.877064 11373 heartbeater.cc:511] Master 127.29.106.62:46573 requested a full tablet report, sending...
I20260811 22:26:05.878865 9487 ts_manager.cc:194] Re-registered known tserver with Master: f16cafe5e19446e782da1ed0fcd4a964 (127.29.106.1:35211)
I20260811 22:26:05.880394 9487 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.106.1@KRBTEST.COM'} at 127.29.106.1:58017
I20260811 22:26:05.969892 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 9862
I20260811 22:26:05.997102 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.2:39773
--local_ip_for_outbound_sockets=127.29.106.2
--tserver_master_addrs=127.29.106.62:46573
--webserver_port=42143
--webserver_interface=127.29.106.2
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.2.keytab
--principal=oryx/127.29.106.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:34763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.29.106.62:46573,127.29.106.61:46755
--tserver_master_addrs=127.29.106.62:46573,127.29.106.61:46755,127.29.106.60:40553 with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:26:06.336342 11393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:26:06.336609 11393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:26:06.336746 11393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:26:06.343737 11393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:26:06.343847 11393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:26:06.343992 11393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.2
I20260811 22:26:06.353286 11393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:34763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.2.keytab
--principal=oryx/127.29.106.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.2:39773
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.106.2
--webserver_port=42143
--tserver_master_addrs=127.29.106.62:46573,127.29.106.61:46755,127.29.106.60:40553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.2
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:26:06.354496 11393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:26:06.355798 11393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:26:06.368603 11401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:06.368530 11405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:06.368752 11402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:26:06.369391 11393 server_base.cc:1034] running on GCE node
I20260811 22:26:06.369997 11393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:26:06.370909 11393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:26:06.372107 11393 hybrid_clock.cc:648] HybridClock initialized: now 1786487166372066 us; error 27 us; skew 500 ppm
I20260811 22:26:06.372391 11393 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:26:06 dist-test-slave-gjrz krb5kdc[9387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487166, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.106.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:26:06.375341 11393 init.cc:377] Logged in from keytab as oryx/127.29.106.2@KRBTEST.COM (short username oryx)
I20260811 22:26:06.376639 11393 webserver.cc:460] Webserver started at http://127.29.106.2:42143/ using document root <none> and password file <none>
I20260811 22:26:06.377235 11393 fs_manager.cc:359] Metadata directory not provided
I20260811 22:26:06.377339 11393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:26:06.382515 11393 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20260811 22:26:06.385164 11419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:06.386377 11393 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 22:26:06.386521 11393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-1/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-1/wal
uuid: "7ff22bba56ee42879d1d18e2630d0fc3"
format_stamp: "Formatted at 2026-08-11 22:26:02 on dist-test-slave-gjrz"
server_key: "e6bd54a9cfb8b483ec8fafe77221ad96"
server_key_iv: "9b4c7b8ffe4f4009cfb490740e8ee3c6"
server_key_version: "encryptionkey@0"
I20260811 22:26:06.387485 11393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:26:06.401711 11393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:26:06.405237 11393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:26:06.405490 11393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:26:06.406386 11393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:26:06.407696 11393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:26:06.407780 11393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:06.407877 11393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:26:06.407948 11393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:06.437978 11393 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.2:39773
I20260811 22:26:06.438045 11638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.2:39773 every 8 connection(s)
I20260811 22:26:06.439477 11393 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
I20260811 22:26:06.449177 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 11393
I20260811 22:26:06.449478 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 10121
Aug 11 22:26:06 dist-test-slave-gjrz krb5kdc[9387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487166, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.106.2@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM
Aug 11 22:26:06 dist-test-slave-gjrz krb5kdc[9387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487166, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.106.2@KRBTEST.COM for oryx/127.29.106.61@KRBTEST.COM
Aug 11 22:26:06 dist-test-slave-gjrz krb5kdc[9387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487166, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.106.2@KRBTEST.COM for oryx/127.29.106.60@KRBTEST.COM
Aug 11 22:26:06 dist-test-slave-gjrz krb5kdc[9387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.106.3@KRBTEST.COM
I20260811 22:26:06.481817 11647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:06.442394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:33325 (local address 127.29.106.62:46573)
0811 22:26:06.453130 (+ 10736us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:06.453137 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:06.459411 (+ 6274us) server_negotiation.cc:407] Connection header received
0811 22:26:06.459602 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:06.459607 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:06.459706 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:06.459805 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:06.461033 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:06.462090 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:06.464765 (+ 2675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:06.465384 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:06.465505 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:06.471497 (+ 5992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:06.471504 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:06.471509 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:06.471552 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:06.476274 (+ 4722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:06.478691 (+ 2417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:06.478696 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:06.478698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:06.478897 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:06.479466 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:06.479471 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:06.479473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:06.479815 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:06.479953 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:06.481000 (+ 1047us) server_negotiation.cc:299] Negotiation successful
0811 22:26:06.481273 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10589,"thread_start_us":14327,"threads_started":1}
I20260811 22:26:06.483480 11644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:06.442310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:44695 (local address 127.29.106.61:46755)
0811 22:26:06.457201 (+ 14891us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:06.457208 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:06.457229 (+ 21us) server_negotiation.cc:407] Connection header received
0811 22:26:06.457299 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:06.457305 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:06.457402 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:06.457508 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:06.460798 (+ 3290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:06.461978 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:06.464712 (+ 2734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:06.465331 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:06.465427 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:06.471890 (+ 6463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:06.471896 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:06.471899 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:06.471930 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:06.477200 (+ 5270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:06.478303 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:06.478308 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:06.478312 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:06.478561 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:06.480076 (+ 1515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:06.480081 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:06.480084 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:06.480590 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:06.480727 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:06.482567 (+ 1840us) server_negotiation.cc:299] Negotiation successful
0811 22:26:06.482781 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14757,"thread_start_us":14644,"threads_started":1}
I20260811 22:26:06.483784 11655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:06.443304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:46573 (local address 127.29.106.2:33325)
0811 22:26:06.459318 (+ 16014us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:06.459337 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:06.459477 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:06.459866 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:06.459870 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:06.460499 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:06.460843 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:06.460857 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:06.462256 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:06.462261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:06.464573 (+ 2312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:06.464589 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:06.465884 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:06.465893 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:06.466266 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:06.467468 (+ 1202us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:06.467504 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:06.471243 (+ 3739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:26:06.477335 (+ 6092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:06.477344 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:06.477361 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:06.478438 (+ 1077us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:06.479017 (+ 579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:06.479022 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:06.479026 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:06.479342 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:06.479954 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:26:06.479962 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:26:06.480479 (+ 517us) client_negotiation.cc:769] Sending connection context
0811 22:26:06.480849 (+ 370us) client_negotiation.cc:240] Negotiation successful
0811 22:26:06.481370 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15874,"mutex_wait_us":146,"spinlock_wait_cycles":33408,"thread_start_us":9994,"threads_started":1}
I20260811 22:26:06.484840 11651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:06.442934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.61:46755 (local address 127.29.106.2:44695)
0811 22:26:06.453681 (+ 10747us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:06.453728 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:06.456387 (+ 2659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:06.458353 (+ 1966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:06.458367 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:06.459176 (+ 809us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:06.460573 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:06.460602 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:06.462183 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:06.462189 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:06.464565 (+ 2376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:06.464578 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:06.465884 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:06.465894 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:06.466260 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:06.467208 (+ 948us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:06.467240 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:06.471677 (+ 4437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:26:06.477334 (+ 5657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:06.477344 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:06.477361 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:06.478148 (+ 787us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:06.479039 (+ 891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:06.479043 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:06.479046 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:06.479380 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:06.480807 (+ 1427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:26:06.482026 (+ 1219us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:26:06.482217 (+ 191us) client_negotiation.cc:769] Sending connection context
0811 22:26:06.483682 (+ 1465us) client_negotiation.cc:240] Negotiation successful
0811 22:26:06.484054 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10459,"mutex_wait_us":645,"thread_start_us":19693,"threads_started":1}
I20260811 22:26:06.485558 11646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:06.442324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:46985 (local address 127.29.106.60:40553)
0811 22:26:06.457998 (+ 15674us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:06.458004 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:06.463188 (+ 5184us) server_negotiation.cc:407] Connection header received
0811 22:26:06.464624 (+ 1436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:06.464629 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:06.464721 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:06.464797 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:06.466145 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:06.467082 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:06.469088 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:06.469743 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:06.469870 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:06.475574 (+ 5704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:06.475581 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:06.475586 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:06.475630 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:06.481056 (+ 5426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:06.482075 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:06.482081 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:06.482084 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:06.482336 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:06.482958 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:06.482963 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:06.482967 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:06.483364 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:06.483551 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:06.484294 (+ 743us) server_negotiation.cc:299] Negotiation successful
0811 22:26:06.484521 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15529,"thread_start_us":15427,"threads_started":1}
I20260811 22:26:06.485891 11641 heartbeater.cc:347] Connected to a master server at 127.29.106.62:46573
I20260811 22:26:06.486263 11641 heartbeater.cc:464] Registering TS with master...
I20260811 22:26:06.487087 11649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:06.442935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:40553 (local address 127.29.106.2:46985)
0811 22:26:06.463087 (+ 20152us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:06.463109 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:06.464458 (+ 1349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:06.464980 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:06.464985 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:06.465507 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:06.465962 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:06.465980 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:06.467250 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:06.467255 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:06.468900 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:06.468915 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:06.470152 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:06.470160 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:06.470347 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:06.471120 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:06.471142 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:06.475290 (+ 4148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:26:06.481227 (+ 5937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:06.481232 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:06.481236 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:06.481939 (+ 703us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:06.482504 (+ 565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:06.482510 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:06.482513 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:06.482807 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:06.483730 (+ 923us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:26:06.483736 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:26:06.483936 (+ 200us) client_negotiation.cc:769] Sending connection context
0811 22:26:06.486048 (+ 2112us) client_negotiation.cc:240] Negotiation successful
0811 22:26:06.486422 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19998,"mutex_wait_us":547,"thread_start_us":16028,"threads_started":1}
I20260811 22:26:06.487669 11641 heartbeater.cc:511] Master 127.29.106.62:46573 requested a full tablet report, sending...
I20260811 22:26:06.488117 11640 heartbeater.cc:347] Connected to a master server at 127.29.106.60:40553
I20260811 22:26:06.488319 11640 heartbeater.cc:464] Registering TS with master...
I20260811 22:26:06.489065 11642 heartbeater.cc:347] Connected to a master server at 127.29.106.61:46755
I20260811 22:26:06.489558 11642 heartbeater.cc:464] Registering TS with master...
I20260811 22:26:06.489776 11640 heartbeater.cc:511] Master 127.29.106.60:40553 requested a full tablet report, sending...
I20260811 22:26:06.490166 9487 ts_manager.cc:194] Re-registered known tserver with Master: 7ff22bba56ee42879d1d18e2630d0fc3 (127.29.106.2:39773)
I20260811 22:26:06.490739 11642 heartbeater.cc:511] Master 127.29.106.61:46755 requested a full tablet report, sending...
I20260811 22:26:06.491250 11657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:06.450527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42274 (local address 127.29.106.3:38811)
0811 22:26:06.462610 (+ 12083us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:06.462619 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:06.462652 (+ 33us) server_negotiation.cc:407] Connection header received
0811 22:26:06.462731 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:06.462739 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:06.462969 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:06.463150 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:06.464769 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:06.466220 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:06.468317 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:06.470001 (+ 1684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:06.470123 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:06.480367 (+ 10244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:06.480383 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:06.480400 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:06.480445 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:06.486572 (+ 6127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:06.487578 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:06.487587 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:06.487590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:06.487953 (+ 363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:06.488616 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:06.488621 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:06.488624 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:06.489143 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:06.489448 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:06.490598 (+ 1150us) server_negotiation.cc:299] Negotiation successful
0811 22:26:06.490846 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11928,"thread_start_us":11773,"threads_started":1}
I20260811 22:26:06.491291 9487 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.106.2@KRBTEST.COM'} at 127.29.106.2:33325
I20260811 22:26:06.491396 11046 ts_manager.cc:194] Registered new tserver with Master: 7ff22bba56ee42879d1d18e2630d0fc3 (127.29.106.2:39773)
I20260811 22:26:06.492797 10364 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:26:06.492832 10653 ts_manager.cc:194] Registered new tserver with Master: 7ff22bba56ee42879d1d18e2630d0fc3 (127.29.106.2:39773)
I20260811 22:26:06.577193 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 10121
I20260811 22:26:06.602566 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.3:38811
--local_ip_for_outbound_sockets=127.29.106.3
--tserver_master_addrs=127.29.106.62:46573
--webserver_port=43601
--webserver_interface=127.29.106.3
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.3.keytab
--principal=oryx/127.29.106.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:34763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.29.106.62:46573,127.29.106.61:46755
--tserver_master_addrs=127.29.106.62:46573,127.29.106.61:46755,127.29.106.60:40553 with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 22:26:06.883656 11373 heartbeater.cc:503] Master 127.29.106.62:46573 was elected leader, sending a full tablet report...
W20260811 22:26:06.928893 11660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:26:06.929140 11660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:26:06.929256 11660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:26:06.936244 11660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:26:06.936334 11660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:26:06.936468 11660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.3
I20260811 22:26:06.945742 11660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:34763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.3.keytab
--principal=oryx/127.29.106.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.3:38811
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.106.3
--webserver_port=43601
--tserver_master_addrs=127.29.106.62:46573,127.29.106.61:46755,127.29.106.60:40553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.3
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:26:06.946926 11660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:26:06.948323 11660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:26:06.962430 11670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:06.962420 11675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:06.962482 11667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:26:06.962941 11660 server_base.cc:1034] running on GCE node
I20260811 22:26:06.963760 11660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:26:06.964730 11660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:26:06.965974 11660 hybrid_clock.cc:648] HybridClock initialized: now 1786487166965929 us; error 31 us; skew 500 ppm
I20260811 22:26:06.966267 11660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:26:06 dist-test-slave-gjrz krb5kdc[9387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487166, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.106.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:26:06.972412 11660 init.cc:377] Logged in from keytab as oryx/127.29.106.3@KRBTEST.COM (short username oryx)
I20260811 22:26:06.973904 11660 webserver.cc:460] Webserver started at http://127.29.106.3:43601/ using document root <none> and password file <none>
I20260811 22:26:06.974526 11660 fs_manager.cc:359] Metadata directory not provided
I20260811 22:26:06.974656 11660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:26:06.979841 11660 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260811 22:26:06.983064 11688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:06.984557 11660 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:26:06.984699 11660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-2/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-2/wal
uuid: "5828483962134e1b972872a8dd583692"
format_stamp: "Formatted at 2026-08-11 22:26:03 on dist-test-slave-gjrz"
server_key: "a48d02f00265cd64e241955f8acf635f"
server_key_iv: "7f0148234c7c86b59fdae66ed9e19f8a"
server_key_version: "encryptionkey@0"
I20260811 22:26:06.985639 11660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:26:07.016546 11660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:26:07.020819 11660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:26:07.021106 11660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:26:07.022138 11660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:26:07.023985 11660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:26:07.024080 11660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:07.024185 11660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:26:07.024264 11660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:07.060065 11660 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.3:38811
I20260811 22:26:07.060087 11901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.3:38811 every 8 connection(s)
I20260811 22:26:07.061596 11660 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
I20260811 22:26:07.064908 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 11660
Aug 11 22:26:07 dist-test-slave-gjrz krb5kdc[9387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487166, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.106.3@KRBTEST.COM for oryx/127.29.106.60@KRBTEST.COM
I20260811 22:26:07.121424 11914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:07.065884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:44393 (local address 127.29.106.60:40553)
0811 22:26:07.096380 (+ 30496us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:07.096388 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:07.096406 (+ 18us) server_negotiation.cc:407] Connection header received
0811 22:26:07.096470 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:07.096474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:07.096585 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:07.096695 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:07.098396 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:07.099557 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:07.102213 (+ 2656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:07.102856 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:07.103921 (+ 1065us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:07.110708 (+ 6787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:07.110716 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:07.110720 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:07.110762 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:07.116775 (+ 6013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:07.117878 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:07.117883 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:07.117886 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:07.118155 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:07.118948 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:07.118952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:07.118955 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:07.119487 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:07.119679 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:07.120559 (+ 880us) server_negotiation.cc:299] Negotiation successful
0811 22:26:07.120793 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30298,"thread_start_us":30934,"threads_started":1}
I20260811 22:26:07.122498 11913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:07.065239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:40553 (local address 127.29.106.3:44393)
0811 22:26:07.082214 (+ 16975us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:07.082272 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:07.085058 (+ 2786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:07.096984 (+ 11926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:07.096994 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:07.097564 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:07.098172 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:07.098190 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:07.099759 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:07.099765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:07.102005 (+ 2240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:07.102021 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:07.103134 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:07.103142 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:07.103471 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:07.104418 (+ 947us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:07.104452 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:07.110429 (+ 5977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:26:07.116917 (+ 6488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:07.116936 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:07.116953 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:07.117721 (+ 768us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:07.118319 (+ 598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:07.118324 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:07.118331 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:07.118803 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:07.119653 (+ 850us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:26:07.119661 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:26:07.120075 (+ 414us) client_negotiation.cc:769] Sending connection context
0811 22:26:07.120401 (+ 326us) client_negotiation.cc:240] Negotiation successful
0811 22:26:07.120921 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16655,"thread_start_us":16364,"threads_started":1}
I20260811 22:26:07.124372 11903 heartbeater.cc:347] Connected to a master server at 127.29.106.60:40553
I20260811 22:26:07.124768 11903 heartbeater.cc:464] Registering TS with master...
I20260811 22:26:07.126477 11903 heartbeater.cc:511] Master 127.29.106.60:40553 requested a full tablet report, sending...
I20260811 22:26:07.128861 11046 ts_manager.cc:194] Registered new tserver with Master: 5828483962134e1b972872a8dd583692 (127.29.106.3:38811)
Aug 11 22:26:07 dist-test-slave-gjrz krb5kdc[9387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487166, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.106.3@KRBTEST.COM for oryx/127.29.106.61@KRBTEST.COM
Aug 11 22:26:07 dist-test-slave-gjrz krb5kdc[9387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487166, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.106.3@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM
I20260811 22:26:07.146637 11920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:07.079205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.61:46755 (local address 127.29.106.3:40601)
0811 22:26:07.109380 (+ 30175us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:07.109415 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:07.109730 (+ 315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:07.110747 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:07.110753 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:07.115355 (+ 4602us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:07.116366 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:07.116380 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:07.118173 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:07.118178 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:07.119469 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:07.119483 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:07.130955 (+ 11472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:07.130968 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:07.131245 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:07.132128 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:07.132153 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:07.135879 (+ 3726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:26:07.142519 (+ 6640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:07.142525 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:07.142529 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:07.143552 (+ 1023us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:07.144074 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:07.144080 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:07.144083 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:07.144377 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:07.145222 (+ 845us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:26:07.145228 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:26:07.145457 (+ 229us) client_negotiation.cc:769] Sending connection context
0811 22:26:07.145599 (+ 142us) client_negotiation.cc:240] Negotiation successful
0811 22:26:07.145962 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29897,"thread_start_us":29181,"threads_started":1}
I20260811 22:26:07.147183 11921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:07.079692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:40601 (local address 127.29.106.61:46755)
0811 22:26:07.106135 (+ 26443us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:07.106142 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:07.109609 (+ 3467us) server_negotiation.cc:407] Connection header received
0811 22:26:07.109913 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:07.109920 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:07.110072 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:07.110169 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:07.116566 (+ 6397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:07.118035 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:07.123389 (+ 5354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:07.124515 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:07.124650 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:07.136184 (+ 11534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:07.136192 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:07.136197 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:07.136275 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:07.142378 (+ 6103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:07.143692 (+ 1314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:07.143697 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:07.143700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:07.143927 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:07.144556 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:07.144561 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:07.144564 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:07.145079 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:07.146093 (+ 1014us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:07.146251 (+ 158us) server_negotiation.cc:299] Negotiation successful
0811 22:26:07.146598 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26261,"thread_start_us":26459,"threads_started":1}
I20260811 22:26:07.148255 11909 heartbeater.cc:347] Connected to a master server at 127.29.106.61:46755
I20260811 22:26:07.148471 11909 heartbeater.cc:464] Registering TS with master...
I20260811 22:26:07.149456 11909 heartbeater.cc:511] Master 127.29.106.61:46755 requested a full tablet report, sending...
I20260811 22:26:07.151067 10653 ts_manager.cc:194] Registered new tserver with Master: 5828483962134e1b972872a8dd583692 (127.29.106.3:38811)
I20260811 22:26:07.157204 11911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:07.064543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:49691 (local address 127.29.106.62:46573)
0811 22:26:07.088431 (+ 23888us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:07.088438 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:07.095807 (+ 7369us) server_negotiation.cc:407] Connection header received
0811 22:26:07.096018 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:07.096041 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:07.096133 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:07.096243 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:07.107237 (+ 10994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:07.109067 (+ 1830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:07.119407 (+ 10340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:07.120329 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:07.120433 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:07.143703 (+ 23270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:07.143713 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:07.143717 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:07.143761 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:07.152029 (+ 8268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:07.153269 (+ 1240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:07.153276 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:07.153279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:07.153572 (+ 293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:07.154131 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:07.154136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:07.154139 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:07.154665 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:07.154817 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:07.156371 (+ 1554us) server_negotiation.cc:299] Negotiation successful
0811 22:26:07.156643 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23753,"thread_start_us":23886,"threads_started":1}
I20260811 22:26:07.157650 11915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:07.066247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:46573 (local address 127.29.106.3:49691)
0811 22:26:07.095700 (+ 29453us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:07.095722 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:07.095868 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:07.099364 (+ 3496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:07.099369 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:07.100044 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:07.100365 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:07.100377 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:07.111252 (+ 10875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:07.111258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:07.115106 (+ 3848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:07.115124 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:07.133556 (+ 18432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:07.133570 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:07.133948 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:07.137308 (+ 3360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:07.137331 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:07.143436 (+ 6105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:26:07.152172 (+ 8736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:07.152175 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:07.152178 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:07.152882 (+ 704us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:07.153727 (+ 845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:07.153732 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:07.153736 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:07.153984 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:07.154895 (+ 911us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:26:07.154911 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:26:07.155137 (+ 226us) client_negotiation.cc:769] Sending connection context
0811 22:26:07.155362 (+ 225us) client_negotiation.cc:240] Negotiation successful
0811 22:26:07.155686 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29318,"thread_start_us":28905,"threads_started":1}
I20260811 22:26:07.159243 11904 heartbeater.cc:347] Connected to a master server at 127.29.106.62:46573
I20260811 22:26:07.159484 11904 heartbeater.cc:464] Registering TS with master...
I20260811 22:26:07.160576 11904 heartbeater.cc:511] Master 127.29.106.62:46573 requested a full tablet report, sending...
I20260811 22:26:07.162259 9487 ts_manager.cc:194] Re-registered known tserver with Master: 5828483962134e1b972872a8dd583692 (127.29.106.3:38811)
I20260811 22:26:07.164254 9487 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.106.3@KRBTEST.COM'} at 127.29.106.3:49691
I20260811 22:26:07.168552 11914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:07.121286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47136 (local address 127.29.106.60:40553)
0811 22:26:07.121662 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:07.121667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:07.137134 (+ 15467us) server_negotiation.cc:407] Connection header received
0811 22:26:07.137438 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:07.137443 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:07.137653 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:07.137747 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:07.143070 (+ 5323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:07.145816 (+ 2746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:07.149521 (+ 3705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:07.152293 (+ 2772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:07.152421 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:07.157980 (+ 5559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:07.157988 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:07.157992 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:07.158035 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:07.163424 (+ 5389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:07.164731 (+ 1307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:07.164737 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:07.164740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:07.165010 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:07.165598 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:07.165603 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:07.165605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:07.165986 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:07.166301 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:07.167671 (+ 1370us) server_negotiation.cc:299] Negotiation successful
0811 22:26:07.167965 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":10368,"threads_started":1}
I20260811 22:26:07.172665 11929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:07.120831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56778 (local address 127.29.106.61:46755)
0811 22:26:07.126022 (+ 5191us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:07.126027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:07.141239 (+ 15212us) server_negotiation.cc:407] Connection header received
0811 22:26:07.141471 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:07.141483 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:07.141559 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:07.141666 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:07.151287 (+ 9621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:07.152959 (+ 1672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:07.155758 (+ 2799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:07.157037 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:07.157219 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:07.162736 (+ 5517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:07.162745 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:07.162751 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:07.162790 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:07.168669 (+ 5879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:07.169566 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:07.169571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:07.169574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:07.169768 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:07.170262 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:07.170267 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:07.170271 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:07.170773 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:07.170988 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:07.171720 (+ 732us) server_negotiation.cc:299] Negotiation successful
0811 22:26:07.171963 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5021,"thread_start_us":4854,"threads_started":1}
I20260811 22:26:07.176302 11928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:07.120572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42378 (local address 127.29.106.62:46573)
0811 22:26:07.132906 (+ 12334us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:07.132913 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:07.143569 (+ 10656us) server_negotiation.cc:407] Connection header received
0811 22:26:07.143649 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:07.143654 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:07.143797 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:07.143931 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:07.152633 (+ 8702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:07.154656 (+ 2023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:07.160039 (+ 5383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:07.161357 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:07.161488 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:07.168256 (+ 6768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:07.168264 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:07.168269 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:07.168308 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:07.173010 (+ 4702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:07.173953 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:07.173957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:07.173959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:07.174305 (+ 346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:07.174627 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:07.174630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:07.174632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:07.174955 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:07.175221 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:07.175581 (+ 360us) server_negotiation.cc:299] Negotiation successful
0811 22:26:07.175763 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12134,"thread_start_us":11789,"threads_started":1}
I20260811 22:26:07.181198 9487 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42378:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:26:07.193986 9487 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:26:07.237290 11955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:07.218184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54002 (local address 127.29.106.2:39773)
0811 22:26:07.219440 (+ 1256us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:07.219451 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:07.225339 (+ 5888us) server_negotiation.cc:407] Connection header received
0811 22:26:07.225464 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:07.225474 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:07.225781 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:07.226014 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:26:07.228546 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:07.230434 (+ 1888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:07.233654 (+ 3220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:07.235724 (+ 2070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:07.235857 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:07.236151 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:07.236589 (+ 438us) server_negotiation.cc:299] Negotiation successful
0811 22:26:07.236826 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":960,"thread_start_us":831,"threads_started":1}
I20260811 22:26:07.237577 11953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:07.217963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.2:39773 (local address 127.0.0.1:54002)
0811 22:26:07.223898 (+ 5935us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:07.223922 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:07.224114 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:07.226321 (+ 2207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:07.226326 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:07.226352 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:26:07.228280 (+ 1928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:07.228299 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:07.230662 (+ 2363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:07.230667 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:07.233504 (+ 2837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:07.233517 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:07.236076 (+ 2559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:07.236085 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:07.236347 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:07.236389 (+ 42us) client_negotiation.cc:769] Sending connection context
0811 22:26:07.236501 (+ 112us) client_negotiation.cc:240] Negotiation successful
0811 22:26:07.236641 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5728,"mutex_wait_us":6,"thread_start_us":5418,"threads_started":1}
I20260811 22:26:07.241760 11509 tablet_service.cc:1467] Processing CreateTablet for tablet 9db8ed87d9fc4b71bf6105330264bf07 (DEFAULT_TABLE table=test-table [id=82a4a99b98b64e4b95e74a4fcee46e15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:26:07.243060 11956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:07.218758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.3:38811 (local address 127.0.0.1:42282)
0811 22:26:07.224800 (+ 6042us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:07.224815 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:07.224941 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:07.228608 (+ 3667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:07.228612 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:07.228639 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:26:07.229806 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:07.229823 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:07.233732 (+ 3909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:07.233738 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:07.238470 (+ 4732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:07.238487 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:07.241606 (+ 3119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:07.241616 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:07.241951 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:07.241991 (+ 40us) client_negotiation.cc:769] Sending connection context
0811 22:26:07.242090 (+ 99us) client_negotiation.cc:240] Negotiation successful
0811 22:26:07.242228 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5865,"mutex_wait_us":2,"thread_start_us":5665,"threads_started":1}
I20260811 22:26:07.243409 11959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:07.224271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40060 (local address 127.29.106.1:35211)
0811 22:26:07.228492 (+ 4221us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:07.228500 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:07.228534 (+ 34us) server_negotiation.cc:407] Connection header received
0811 22:26:07.228630 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:07.228638 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:07.228907 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:07.229141 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:26:07.231290 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:07.233521 (+ 2231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:07.238962 (+ 5441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:07.241676 (+ 2714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:07.241794 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:07.242093 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:07.242538 (+ 445us) server_negotiation.cc:299] Negotiation successful
0811 22:26:07.242776 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3989,"thread_start_us":3794,"threads_started":1}
I20260811 22:26:07.243672 11509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9db8ed87d9fc4b71bf6105330264bf07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:26:07.243947 11958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:07.219413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42282 (local address 127.29.106.3:38811)
0811 22:26:07.227898 (+ 8485us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:07.227907 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:07.227978 (+ 71us) server_negotiation.cc:407] Connection header received
0811 22:26:07.228080 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:07.228087 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:07.228387 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:07.228608 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:26:07.230815 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:07.233596 (+ 2781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:07.238693 (+ 5097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:07.241234 (+ 2541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:07.241387 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:07.241625 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:07.242765 (+ 1140us) server_negotiation.cc:299] Negotiation successful
0811 22:26:07.242972 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8270,"thread_start_us":7865,"threads_started":1}
I20260811 22:26:07.244978 11960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:07.225129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:35211 (local address 127.0.0.1:40060)
0811 22:26:07.226268 (+ 1139us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:07.226285 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:07.226433 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:07.229266 (+ 2833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:07.229271 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:07.229297 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:26:07.231037 (+ 1740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:07.231052 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:07.233727 (+ 2675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:07.233732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:07.238765 (+ 5033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:07.238780 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:07.242123 (+ 3343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:07.242133 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:07.242333 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:07.242368 (+ 35us) client_negotiation.cc:769] Sending connection context
0811 22:26:07.242461 (+ 93us) client_negotiation.cc:240] Negotiation successful
0811 22:26:07.244120 (+ 1659us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":977,"mutex_wait_us":44,"thread_start_us":1259,"threads_started":1}
I20260811 22:26:07.248237 11786 tablet_service.cc:1467] Processing CreateTablet for tablet 9db8ed87d9fc4b71bf6105330264bf07 (DEFAULT_TABLE table=test-table [id=82a4a99b98b64e4b95e74a4fcee46e15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:26:07.249868 11786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9db8ed87d9fc4b71bf6105330264bf07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:26:07.249500 11242 tablet_service.cc:1467] Processing CreateTablet for tablet 9db8ed87d9fc4b71bf6105330264bf07 (DEFAULT_TABLE table=test-table [id=82a4a99b98b64e4b95e74a4fcee46e15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:26:07.251339 11242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9db8ed87d9fc4b71bf6105330264bf07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:26:07.268170 11961 tablet_bootstrap.cc:492] T 9db8ed87d9fc4b71bf6105330264bf07 P 5828483962134e1b972872a8dd583692: Bootstrap starting.
I20260811 22:26:07.274365 11961 tablet_bootstrap.cc:654] T 9db8ed87d9fc4b71bf6105330264bf07 P 5828483962134e1b972872a8dd583692: Neither blocks nor log segments found. Creating new log.
I20260811 22:26:07.276364 11961 log.cc:824] T 9db8ed87d9fc4b71bf6105330264bf07 P 5828483962134e1b972872a8dd583692: Log is configured to *not* fsync() on all Append() calls
I20260811 22:26:07.277985 11962 tablet_bootstrap.cc:492] T 9db8ed87d9fc4b71bf6105330264bf07 P 7ff22bba56ee42879d1d18e2630d0fc3: Bootstrap starting.
I20260811 22:26:07.284794 11962 tablet_bootstrap.cc:654] T 9db8ed87d9fc4b71bf6105330264bf07 P 7ff22bba56ee42879d1d18e2630d0fc3: Neither blocks nor log segments found. Creating new log.
I20260811 22:26:07.284948 11961 tablet_bootstrap.cc:492] T 9db8ed87d9fc4b71bf6105330264bf07 P 5828483962134e1b972872a8dd583692: No bootstrap required, opened a new log
I20260811 22:26:07.285229 11961 ts_tablet_manager.cc:1397] T 9db8ed87d9fc4b71bf6105330264bf07 P 5828483962134e1b972872a8dd583692: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.009s
I20260811 22:26:07.288820 11962 log.cc:824] T 9db8ed87d9fc4b71bf6105330264bf07 P 7ff22bba56ee42879d1d18e2630d0fc3: Log is configured to *not* fsync() on all Append() calls
I20260811 22:26:07.291008 11966 tablet_bootstrap.cc:492] T 9db8ed87d9fc4b71bf6105330264bf07 P f16cafe5e19446e782da1ed0fcd4a964: Bootstrap starting.
I20260811 22:26:07.297273 11966 tablet_bootstrap.cc:654] T 9db8ed87d9fc4b71bf6105330264bf07 P f16cafe5e19446e782da1ed0fcd4a964: Neither blocks nor log segments found. Creating new log.
I20260811 22:26:07.299682 11962 tablet_bootstrap.cc:492] T 9db8ed87d9fc4b71bf6105330264bf07 P 7ff22bba56ee42879d1d18e2630d0fc3: No bootstrap required, opened a new log
I20260811 22:26:07.299755 11966 log.cc:824] T 9db8ed87d9fc4b71bf6105330264bf07 P f16cafe5e19446e782da1ed0fcd4a964: Log is configured to *not* fsync() on all Append() calls
I20260811 22:26:07.300246 11962 ts_tablet_manager.cc:1397] T 9db8ed87d9fc4b71bf6105330264bf07 P 7ff22bba56ee42879d1d18e2630d0fc3: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.014s
I20260811 22:26:07.303988 11966 tablet_bootstrap.cc:492] T 9db8ed87d9fc4b71bf6105330264bf07 P f16cafe5e19446e782da1ed0fcd4a964: No bootstrap required, opened a new log
I20260811 22:26:07.304288 11966 ts_tablet_manager.cc:1397] T 9db8ed87d9fc4b71bf6105330264bf07 P f16cafe5e19446e782da1ed0fcd4a964: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.008s
I20260811 22:26:07.305639 11961 raft_consensus.cc:348] T 9db8ed87d9fc4b71bf6105330264bf07 P 5828483962134e1b972872a8dd583692 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff22bba56ee42879d1d18e2630d0fc3" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39773 } } peers { permanent_uuid: "f16cafe5e19446e782da1ed0fcd4a964" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 35211 } } peers { permanent_uuid: "5828483962134e1b972872a8dd583692" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38811 } }
I20260811 22:26:07.306317 11961 raft_consensus.cc:374] T 9db8ed87d9fc4b71bf6105330264bf07 P 5828483962134e1b972872a8dd583692 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:26:07.306514 11961 raft_consensus.cc:729] T 9db8ed87d9fc4b71bf6105330264bf07 P 5828483962134e1b972872a8dd583692 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5828483962134e1b972872a8dd583692, State: Initialized, Role: FOLLOWER
I20260811 22:26:07.307377 11961 consensus_queue.cc:260] T 9db8ed87d9fc4b71bf6105330264bf07 P 5828483962134e1b972872a8dd583692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff22bba56ee42879d1d18e2630d0fc3" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39773 } } peers { permanent_uuid: "f16cafe5e19446e782da1ed0fcd4a964" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 35211 } } peers { permanent_uuid: "5828483962134e1b972872a8dd583692" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38811 } }
I20260811 22:26:07.313632 11961 ts_tablet_manager.cc:1428] T 9db8ed87d9fc4b71bf6105330264bf07 P 5828483962134e1b972872a8dd583692: Time spent starting tablet: real 0.028s user 0.016s sys 0.011s
I20260811 22:26:07.314065 11904 heartbeater.cc:503] Master 127.29.106.62:46573 was elected leader, sending a full tablet report...
I20260811 22:26:07.319190 11966 raft_consensus.cc:348] T 9db8ed87d9fc4b71bf6105330264bf07 P f16cafe5e19446e782da1ed0fcd4a964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff22bba56ee42879d1d18e2630d0fc3" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39773 } } peers { permanent_uuid: "f16cafe5e19446e782da1ed0fcd4a964" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 35211 } } peers { permanent_uuid: "5828483962134e1b972872a8dd583692" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38811 } }
I20260811 22:26:07.319572 11966 raft_consensus.cc:374] T 9db8ed87d9fc4b71bf6105330264bf07 P f16cafe5e19446e782da1ed0fcd4a964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:26:07.319756 11966 raft_consensus.cc:729] T 9db8ed87d9fc4b71bf6105330264bf07 P f16cafe5e19446e782da1ed0fcd4a964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f16cafe5e19446e782da1ed0fcd4a964, State: Initialized, Role: FOLLOWER
I20260811 22:26:07.320465 11966 consensus_queue.cc:260] T 9db8ed87d9fc4b71bf6105330264bf07 P f16cafe5e19446e782da1ed0fcd4a964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff22bba56ee42879d1d18e2630d0fc3" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39773 } } peers { permanent_uuid: "f16cafe5e19446e782da1ed0fcd4a964" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 35211 } } peers { permanent_uuid: "5828483962134e1b972872a8dd583692" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38811 } }
I20260811 22:26:07.322361 11962 raft_consensus.cc:348] T 9db8ed87d9fc4b71bf6105330264bf07 P 7ff22bba56ee42879d1d18e2630d0fc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff22bba56ee42879d1d18e2630d0fc3" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39773 } } peers { permanent_uuid: "f16cafe5e19446e782da1ed0fcd4a964" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 35211 } } peers { permanent_uuid: "5828483962134e1b972872a8dd583692" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38811 } }
I20260811 22:26:07.322981 11966 ts_tablet_manager.cc:1428] T 9db8ed87d9fc4b71bf6105330264bf07 P f16cafe5e19446e782da1ed0fcd4a964: Time spent starting tablet: real 0.019s user 0.012s sys 0.007s
I20260811 22:26:07.323033 11962 raft_consensus.cc:374] T 9db8ed87d9fc4b71bf6105330264bf07 P 7ff22bba56ee42879d1d18e2630d0fc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:26:07.323222 11962 raft_consensus.cc:729] T 9db8ed87d9fc4b71bf6105330264bf07 P 7ff22bba56ee42879d1d18e2630d0fc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ff22bba56ee42879d1d18e2630d0fc3, State: Initialized, Role: FOLLOWER
I20260811 22:26:07.324047 11962 consensus_queue.cc:260] T 9db8ed87d9fc4b71bf6105330264bf07 P 7ff22bba56ee42879d1d18e2630d0fc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff22bba56ee42879d1d18e2630d0fc3" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39773 } } peers { permanent_uuid: "f16cafe5e19446e782da1ed0fcd4a964" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 35211 } } peers { permanent_uuid: "5828483962134e1b972872a8dd583692" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38811 } }
I20260811 22:26:07.328362 11641 heartbeater.cc:503] Master 127.29.106.62:46573 was elected leader, sending a full tablet report...
I20260811 22:26:07.330121 11962 ts_tablet_manager.cc:1428] T 9db8ed87d9fc4b71bf6105330264bf07 P 7ff22bba56ee42879d1d18e2630d0fc3: Time spent starting tablet: real 0.030s user 0.015s sys 0.008s
W20260811 22:26:07.368444 11919 tablet.cc:2369] T 9db8ed87d9fc4b71bf6105330264bf07 P 5828483962134e1b972872a8dd583692: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 22:26:07.442848 11643 tablet.cc:2369] T 9db8ed87d9fc4b71bf6105330264bf07 P 7ff22bba56ee42879d1d18e2630d0fc3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 22:26:07.572518 11375 tablet.cc:2369] T 9db8ed87d9fc4b71bf6105330264bf07 P f16cafe5e19446e782da1ed0fcd4a964: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:26:07.626518 11974 raft_consensus.cc:482] T 9db8ed87d9fc4b71bf6105330264bf07 P f16cafe5e19446e782da1ed0fcd4a964 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:26:07.627075 11974 raft_consensus.cc:504] T 9db8ed87d9fc4b71bf6105330264bf07 P f16cafe5e19446e782da1ed0fcd4a964 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff22bba56ee42879d1d18e2630d0fc3" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39773 } } peers { permanent_uuid: "f16cafe5e19446e782da1ed0fcd4a964" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 35211 } } peers { permanent_uuid: "5828483962134e1b972872a8dd583692" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38811 } }
I20260811 22:26:07.629648 11974 leader_election.cc:288] T 9db8ed87d9fc4b71bf6105330264bf07 P f16cafe5e19446e782da1ed0fcd4a964 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ff22bba56ee42879d1d18e2630d0fc3 (127.29.106.2:39773), 5828483962134e1b972872a8dd583692 (127.29.106.3:38811)
I20260811 22:26:07.640362 11958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:07.630057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:35553 (local address 127.29.106.3:38811)
0811 22:26:07.630244 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:07.630249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:07.632179 (+ 1930us) server_negotiation.cc:407] Connection header received
0811 22:26:07.632406 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:07.632411 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:07.632471 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:07.632546 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:26:07.633839 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:07.634587 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:07.637965 (+ 3378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:07.639230 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:07.639475 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:07.639574 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:07.639932 (+ 358us) server_negotiation.cc:299] Negotiation successful
0811 22:26:07.640055 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260811 22:26:07.640892 11955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:07.629917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:47281 (local address 127.29.106.2:39773)
0811 22:26:07.630131 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:07.630138 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:07.631999 (+ 1861us) server_negotiation.cc:407] Connection header received
0811 22:26:07.632335 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:07.632341 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:07.632408 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:07.632482 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:26:07.633688 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:07.634527 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:07.638047 (+ 3520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:07.639505 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:07.639647 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:07.639725 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:07.640250 (+ 525us) server_negotiation.cc:299] Negotiation successful
0811 22:26:07.640369 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260811 22:26:07.640935 11979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:07.629761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.3:38811 (local address 127.29.106.1:35553)
0811 22:26:07.632003 (+ 2242us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:07.632062 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:07.632251 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:07.632575 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:07.632580 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:07.632603 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:26:07.633661 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:07.633674 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:07.634760 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:07.634764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:07.637774 (+ 3010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:07.637787 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:07.639592 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:07.639600 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:07.639772 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:07.639797 (+ 25us) client_negotiation.cc:769] Sending connection context
0811 22:26:07.639891 (+ 94us) client_negotiation.cc:240] Negotiation successful
0811 22:26:07.640016 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2078,"mutex_wait_us":143,"thread_start_us":1564,"threads_started":1}
I20260811 22:26:07.641222 11978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:07.629761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.2:39773 (local address 127.29.106.1:47281)
0811 22:26:07.631887 (+ 2126us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:07.631904 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:07.632100 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:07.632514 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:07.632518 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:07.632587 (+ 69us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:26:07.633537 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:07.633551 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:07.634708 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:07.634711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:07.637834 (+ 3123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:07.637847 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:07.639882 (+ 2035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:07.639888 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:07.640096 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:07.640120 (+ 24us) client_negotiation.cc:769] Sending connection context
0811 22:26:07.640203 (+ 83us) client_negotiation.cc:240] Negotiation successful
0811 22:26:07.640325 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1911,"mutex_wait_us":55,"thread_start_us":1128,"threads_started":1}
I20260811 22:26:07.642445 11821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9db8ed87d9fc4b71bf6105330264bf07" candidate_uuid: "f16cafe5e19446e782da1ed0fcd4a964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5828483962134e1b972872a8dd583692" is_pre_election: true
I20260811 22:26:07.642566 11548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9db8ed87d9fc4b71bf6105330264bf07" candidate_uuid: "f16cafe5e19446e782da1ed0fcd4a964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ff22bba56ee42879d1d18e2630d0fc3" is_pre_election: true
I20260811 22:26:07.643085 11821 raft_consensus.cc:2454] T 9db8ed87d9fc4b71bf6105330264bf07 P 5828483962134e1b972872a8dd583692 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f16cafe5e19446e782da1ed0fcd4a964 in term 0.
I20260811 22:26:07.643404 11548 raft_consensus.cc:2454] T 9db8ed87d9fc4b71bf6105330264bf07 P 7ff22bba56ee42879d1d18e2630d0fc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f16cafe5e19446e782da1ed0fcd4a964 in term 0.
I20260811 22:26:07.643874 11158 leader_election.cc:302] T 9db8ed87d9fc4b71bf6105330264bf07 P f16cafe5e19446e782da1ed0fcd4a964 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5828483962134e1b972872a8dd583692, f16cafe5e19446e782da1ed0fcd4a964; no voters:
I20260811 22:26:07.644340 11974 raft_consensus.cc:2781] T 9db8ed87d9fc4b71bf6105330264bf07 P f16cafe5e19446e782da1ed0fcd4a964 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:26:07.644557 11974 raft_consensus.cc:482] T 9db8ed87d9fc4b71bf6105330264bf07 P f16cafe5e19446e782da1ed0fcd4a964 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:26:07.644711 11974 raft_consensus.cc:3037] T 9db8ed87d9fc4b71bf6105330264bf07 P f16cafe5e19446e782da1ed0fcd4a964 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:26:07.648257 11974 raft_consensus.cc:504] T 9db8ed87d9fc4b71bf6105330264bf07 P f16cafe5e19446e782da1ed0fcd4a964 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff22bba56ee42879d1d18e2630d0fc3" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39773 } } peers { permanent_uuid: "f16cafe5e19446e782da1ed0fcd4a964" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 35211 } } peers { permanent_uuid: "5828483962134e1b972872a8dd583692" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38811 } }
I20260811 22:26:07.649233 11974 leader_election.cc:288] T 9db8ed87d9fc4b71bf6105330264bf07 P f16cafe5e19446e782da1ed0fcd4a964 [CANDIDATE]: Term 1 election: Requested vote from peers 7ff22bba56ee42879d1d18e2630d0fc3 (127.29.106.2:39773), 5828483962134e1b972872a8dd583692 (127.29.106.3:38811)
I20260811 22:26:07.649449 11548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9db8ed87d9fc4b71bf6105330264bf07" candidate_uuid: "f16cafe5e19446e782da1ed0fcd4a964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ff22bba56ee42879d1d18e2630d0fc3"
I20260811 22:26:07.649631 11548 raft_consensus.cc:3037] T 9db8ed87d9fc4b71bf6105330264bf07 P 7ff22bba56ee42879d1d18e2630d0fc3 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:26:07.649761 11821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9db8ed87d9fc4b71bf6105330264bf07" candidate_uuid: "f16cafe5e19446e782da1ed0fcd4a964" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5828483962134e1b972872a8dd583692"
I20260811 22:26:07.650085 11821 raft_consensus.cc:3037] T 9db8ed87d9fc4b71bf6105330264bf07 P 5828483962134e1b972872a8dd583692 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:26:07.653497 11548 raft_consensus.cc:2454] T 9db8ed87d9fc4b71bf6105330264bf07 P 7ff22bba56ee42879d1d18e2630d0fc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f16cafe5e19446e782da1ed0fcd4a964 in term 1.
I20260811 22:26:07.653772 11821 raft_consensus.cc:2454] T 9db8ed87d9fc4b71bf6105330264bf07 P 5828483962134e1b972872a8dd583692 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f16cafe5e19446e782da1ed0fcd4a964 in term 1.
I20260811 22:26:07.653874 11162 leader_election.cc:302] T 9db8ed87d9fc4b71bf6105330264bf07 P f16cafe5e19446e782da1ed0fcd4a964 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ff22bba56ee42879d1d18e2630d0fc3, f16cafe5e19446e782da1ed0fcd4a964; no voters:
I20260811 22:26:07.654332 11974 raft_consensus.cc:2781] T 9db8ed87d9fc4b71bf6105330264bf07 P f16cafe5e19446e782da1ed0fcd4a964 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:26:07.655253 11974 raft_consensus.cc:686] T 9db8ed87d9fc4b71bf6105330264bf07 P f16cafe5e19446e782da1ed0fcd4a964 [term 1 LEADER]: Becoming Leader. State: Replica: f16cafe5e19446e782da1ed0fcd4a964, State: Running, Role: LEADER
I20260811 22:26:07.656093 11974 consensus_queue.cc:237] T 9db8ed87d9fc4b71bf6105330264bf07 P f16cafe5e19446e782da1ed0fcd4a964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff22bba56ee42879d1d18e2630d0fc3" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39773 } } peers { permanent_uuid: "f16cafe5e19446e782da1ed0fcd4a964" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 35211 } } peers { permanent_uuid: "5828483962134e1b972872a8dd583692" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38811 } }
I20260811 22:26:07.662128 9487 catalog_manager.cc:5495] T 9db8ed87d9fc4b71bf6105330264bf07 P f16cafe5e19446e782da1ed0fcd4a964 reported cstate change: term changed from 0 to 1, leader changed from <none> to f16cafe5e19446e782da1ed0fcd4a964 (127.29.106.1). New cstate: current_term: 1 leader_uuid: "f16cafe5e19446e782da1ed0fcd4a964" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff22bba56ee42879d1d18e2630d0fc3" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f16cafe5e19446e782da1ed0fcd4a964" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 35211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5828483962134e1b972872a8dd583692" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38811 } health_report { overall_health: UNKNOWN } } }
I20260811 22:26:07.686995 11959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:07.681122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40066 (local address 127.29.106.1:35211)
0811 22:26:07.681331 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:07.681337 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:07.681567 (+ 230us) server_negotiation.cc:407] Connection header received
0811 22:26:07.681761 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:07.681765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:07.681844 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:07.681984 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:26:07.682383 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:07.683134 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:07.684921 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:07.685325 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:07.685434 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:07.685897 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:26:07.686372 (+ 475us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:26:07.686502 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:07.686612 (+ 110us) server_negotiation.cc:299] Negotiation successful
0811 22:26:07.686695 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 22:26:07.698195 11821 raft_consensus.cc:1261] T 9db8ed87d9fc4b71bf6105330264bf07 P 5828483962134e1b972872a8dd583692 [term 1 FOLLOWER]: Refusing update from remote peer f16cafe5e19446e782da1ed0fcd4a964: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:26:07.698598 11548 raft_consensus.cc:1261] T 9db8ed87d9fc4b71bf6105330264bf07 P 7ff22bba56ee42879d1d18e2630d0fc3 [term 1 FOLLOWER]: Refusing update from remote peer f16cafe5e19446e782da1ed0fcd4a964: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:26:07.699365 11980 consensus_queue.cc:1035] T 9db8ed87d9fc4b71bf6105330264bf07 P f16cafe5e19446e782da1ed0fcd4a964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5828483962134e1b972872a8dd583692" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260811 22:26:07.700194 11974 consensus_queue.cc:1035] T 9db8ed87d9fc4b71bf6105330264bf07 P f16cafe5e19446e782da1ed0fcd4a964 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ff22bba56ee42879d1d18e2630d0fc3" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:26:07.715070 11986 mvcc.cc:205] Tried to move back new op lower bound from 7317451438881689600 to 7317451438723588096. Current Snapshot: MvccSnapshot[applied={T|T < 7317451438881689600}]
I20260811 22:26:07.718618 11988 mvcc.cc:205] Tried to move back new op lower bound from 7317451438881689600 to 7317451438723588096. Current Snapshot: MvccSnapshot[applied={T|T < 7317451438881689600}]
I20260811 22:26:07.720943 11989 mvcc.cc:205] Tried to move back new op lower bound from 7317451438881689600 to 7317451438723588096. Current Snapshot: MvccSnapshot[applied={T|T < 7317451438881689600}]
I20260811 22:26:07.738993 9484 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:26:07.739389 9484 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:26:07.746100 9484 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b5d1ed0b161417688f43a878b41ef3d: Sending DeleteTablet for 3 replicas of tablet 9db8ed87d9fc4b71bf6105330264bf07
I20260811 22:26:07.747440 11509 tablet_service.cc:1514] Processing DeleteTablet for tablet 9db8ed87d9fc4b71bf6105330264bf07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:26:07 UTC) from {username='oryx', principal='oryx/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:54002
I20260811 22:26:07.747720 11242 tablet_service.cc:1514] Processing DeleteTablet for tablet 9db8ed87d9fc4b71bf6105330264bf07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:26:07 UTC) from {username='oryx', principal='oryx/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:40060
I20260811 22:26:07.748903 11786 tablet_service.cc:1514] Processing DeleteTablet for tablet 9db8ed87d9fc4b71bf6105330264bf07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:26:07 UTC) from {username='oryx', principal='oryx/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:42282
I20260811 22:26:07.756880 12012 tablet_replica.cc:330] T 9db8ed87d9fc4b71bf6105330264bf07 P 5828483962134e1b972872a8dd583692: stopping tablet replica
I20260811 22:26:07.757555 12010 tablet_replica.cc:330] T 9db8ed87d9fc4b71bf6105330264bf07 P 7ff22bba56ee42879d1d18e2630d0fc3: stopping tablet replica
I20260811 22:26:07.757553 12012 raft_consensus.cc:2229] T 9db8ed87d9fc4b71bf6105330264bf07 P 5828483962134e1b972872a8dd583692 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:26:07.758104 12010 raft_consensus.cc:2229] T 9db8ed87d9fc4b71bf6105330264bf07 P 7ff22bba56ee42879d1d18e2630d0fc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:26:07.759758 12012 raft_consensus.cc:2258] T 9db8ed87d9fc4b71bf6105330264bf07 P 5828483962134e1b972872a8dd583692 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:26:07.764464 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 9418
I20260811 22:26:07.765650 9540 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:26:07.767251 12012 ts_tablet_manager.cc:1905] T 9db8ed87d9fc4b71bf6105330264bf07 P 5828483962134e1b972872a8dd583692: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:26:07.767699 12011 tablet_replica.cc:330] T 9db8ed87d9fc4b71bf6105330264bf07 P f16cafe5e19446e782da1ed0fcd4a964: stopping tablet replica
I20260811 22:26:07.768864 12011 raft_consensus.cc:2229] T 9db8ed87d9fc4b71bf6105330264bf07 P f16cafe5e19446e782da1ed0fcd4a964 [term 1 LEADER]: Raft consensus shutting down.
I20260811 22:26:07.769897 12011 raft_consensus.cc:2258] T 9db8ed87d9fc4b71bf6105330264bf07 P f16cafe5e19446e782da1ed0fcd4a964 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:26:07.773802 12010 raft_consensus.cc:2258] T 9db8ed87d9fc4b71bf6105330264bf07 P 7ff22bba56ee42879d1d18e2630d0fc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:26:07.775019 12011 ts_tablet_manager.cc:1905] T 9db8ed87d9fc4b71bf6105330264bf07 P f16cafe5e19446e782da1ed0fcd4a964: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:26:07.776247 12010 ts_tablet_manager.cc:1905] T 9db8ed87d9fc4b71bf6105330264bf07 P 7ff22bba56ee42879d1d18e2630d0fc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:26:07.782819 12012 ts_tablet_manager.cc:1918] T 9db8ed87d9fc4b71bf6105330264bf07 P 5828483962134e1b972872a8dd583692: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:26:07.783074 12012 log.cc:1196] T 9db8ed87d9fc4b71bf6105330264bf07 P 5828483962134e1b972872a8dd583692: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-2/wal/wals/9db8ed87d9fc4b71bf6105330264bf07
I20260811 22:26:07.783887 12012 ts_tablet_manager.cc:1939] T 9db8ed87d9fc4b71bf6105330264bf07 P 5828483962134e1b972872a8dd583692: Deleting consensus metadata
I20260811 22:26:07.786573 12010 ts_tablet_manager.cc:1918] T 9db8ed87d9fc4b71bf6105330264bf07 P 7ff22bba56ee42879d1d18e2630d0fc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:26:07.786924 12010 log.cc:1196] T 9db8ed87d9fc4b71bf6105330264bf07 P 7ff22bba56ee42879d1d18e2630d0fc3: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-1/wal/wals/9db8ed87d9fc4b71bf6105330264bf07
I20260811 22:26:07.787256 12015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:07.752170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56782 (local address 127.29.106.61:46755)
0811 22:26:07.759767 (+ 7597us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:07.759775 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:07.761137 (+ 1362us) server_negotiation.cc:407] Connection header received
0811 22:26:07.761198 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:07.761204 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:07.761287 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:07.761381 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:07.765458 (+ 4077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:07.767406 (+ 1948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:07.770103 (+ 2697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:07.771672 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:07.771770 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:07.777332 (+ 5562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:07.777340 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:07.777345 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:07.777383 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:07.782924 (+ 5541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:07.784102 (+ 1178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:07.784108 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:07.784112 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:07.784348 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:07.784882 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:07.784887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:07.784890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:07.785386 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:07.785767 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:07.786473 (+ 706us) server_negotiation.cc:299] Negotiation successful
0811 22:26:07.786749 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7432,"thread_start_us":7087,"threads_started":1}
I20260811 22:26:07.788177 12010 ts_tablet_manager.cc:1939] T 9db8ed87d9fc4b71bf6105330264bf07 P 7ff22bba56ee42879d1d18e2630d0fc3: Deleting consensus metadata
I20260811 22:26:07.797230 12011 ts_tablet_manager.cc:1918] T 9db8ed87d9fc4b71bf6105330264bf07 P f16cafe5e19446e782da1ed0fcd4a964: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:26:07.797569 12011 log.cc:1196] T 9db8ed87d9fc4b71bf6105330264bf07 P f16cafe5e19446e782da1ed0fcd4a964: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-0/wal/wals/9db8ed87d9fc4b71bf6105330264bf07
I20260811 22:26:07.798597 12011 ts_tablet_manager.cc:1939] T 9db8ed87d9fc4b71bf6105330264bf07 P f16cafe5e19446e782da1ed0fcd4a964: Deleting consensus metadata
I20260811 22:26:07.814378 12016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:07.760822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47138 (local address 127.29.106.60:40553)
0811 22:26:07.768402 (+ 7580us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:07.768408 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:07.788316 (+ 19908us) server_negotiation.cc:407] Connection header received
0811 22:26:07.788586 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:07.788592 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:07.788719 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:07.788806 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:07.791919 (+ 3113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:07.794070 (+ 2151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:07.796925 (+ 2855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:07.798075 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:07.798183 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:07.803847 (+ 5664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:07.803855 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:07.803859 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:07.803897 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:07.809667 (+ 5770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:07.810851 (+ 1184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:07.810858 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:07.810862 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:07.811221 (+ 359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:07.811906 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:07.811912 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:07.811916 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:07.812445 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:07.812807 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:07.813506 (+ 699us) server_negotiation.cc:299] Negotiation successful
0811 22:26:07.813782 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7425,"thread_start_us":7284,"threads_started":1}
I20260811 22:26:07.919770 9453 catalog_manager.cc:4841] TS 5828483962134e1b972872a8dd583692 (127.29.106.3:38811): tablet 9db8ed87d9fc4b71bf6105330264bf07 (table test-table [id=82a4a99b98b64e4b95e74a4fcee46e15]) successfully deleted
I20260811 22:26:07.920467 9455 catalog_manager.cc:4841] TS f16cafe5e19446e782da1ed0fcd4a964 (127.29.106.1:35211): tablet 9db8ed87d9fc4b71bf6105330264bf07 (table test-table [id=82a4a99b98b64e4b95e74a4fcee46e15]) successfully deleted
I20260811 22:26:07.920746 9455 catalog_manager.cc:4841] TS 7ff22bba56ee42879d1d18e2630d0fc3 (127.29.106.2:39773): tablet 9db8ed87d9fc4b71bf6105330264bf07 (table test-table [id=82a4a99b98b64e4b95e74a4fcee46e15]) successfully deleted
I20260811 22:26:07.921293 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 9418
I20260811 22:26:07.980809 12015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:07.956395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56794 (local address 127.29.106.61:46755)
0811 22:26:07.956546 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:07.956551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:07.964276 (+ 7725us) server_negotiation.cc:407] Connection header received
0811 22:26:07.964592 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:07.964599 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:07.964686 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:07.964757 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:07.967653 (+ 2896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:07.968835 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:07.970307 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:07.970791 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:07.970905 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:07.974154 (+ 3249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:07.974161 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:07.974165 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:07.974199 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:07.977720 (+ 3521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:07.978457 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:07.978486 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:07.978490 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:07.978658 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:07.979075 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:07.979080 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:07.979084 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:07.979427 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:07.979679 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:07.980217 (+ 538us) server_negotiation.cc:299] Negotiation successful
0811 22:26:07.980578 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260811 22:26:07.984192 12016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:07.956166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47148 (local address 127.29.106.60:40553)
0811 22:26:07.956332 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:07.956338 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:07.968307 (+ 11969us) server_negotiation.cc:407] Connection header received
0811 22:26:07.968849 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:07.968854 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:07.968926 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:07.969024 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:07.971416 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:07.972408 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:07.974380 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:07.974963 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:07.975103 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:07.978095 (+ 2992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:07.978101 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:07.978106 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:07.978147 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:07.981542 (+ 3395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:07.982433 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:07.982437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:07.982440 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:07.982581 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:07.982951 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:07.982954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:07.982956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:07.983215 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:07.983481 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:07.983783 (+ 302us) server_negotiation.cc:299] Negotiation successful
0811 22:26:07.983959 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260811 22:26:08.727075 12028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:08.726396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:46573 (local address 127.29.106.3:34003)
0811 22:26:08.726799 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:08.726915 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":208,"threads_started":1}
W20260811 22:26:08.727573 11904 heartbeater.cc:644] Failed to heartbeat to 127.29.106.62:46573 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.29.106.62:46573: connect: Connection refused (error 111)
I20260811 22:26:08.728726 12029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:08.727950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:46573 (local address 127.29.106.2:52223)
0811 22:26:08.728403 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:08.728540 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":226,"threads_started":1}
W20260811 22:26:08.729254 11641 heartbeater.cc:644] Failed to heartbeat to 127.29.106.62:46573 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.29.106.62:46573: connect: Connection refused (error 111)
I20260811 22:26:08.736039 12030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:08.735199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:46573 (local address 127.29.106.1:49045)
0811 22:26:08.735654 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:08.735810 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":243,"threads_started":1}
W20260811 22:26:08.736392 11373 heartbeater.cc:644] Failed to heartbeat to 127.29.106.62:46573 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.29.106.62:46573: connect: Connection refused (error 111)
I20260811 22:26:09.247483 12031 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2b5d1ed0b161417688f43a878b41ef3d)
I20260811 22:26:09.247658 12031 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: false } } peers { permanent_uuid: "800b0066b8d34ef98b9c2bc24409ee79" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 40553 } attrs { promote: false } }
I20260811 22:26:09.247987 12037 raft_consensus.cc:482] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2b5d1ed0b161417688f43a878b41ef3d)
I20260811 22:26:09.248159 12037 raft_consensus.cc:504] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: false } } peers { permanent_uuid: "800b0066b8d34ef98b9c2bc24409ee79" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 40553 } attrs { promote: false } }
I20260811 22:26:09.248476 12031 leader_election.cc:288] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b5d1ed0b161417688f43a878b41ef3d (127.29.106.62:46573), 800b0066b8d34ef98b9c2bc24409ee79 (127.29.106.60:40553)
I20260811 22:26:09.249001 12037 leader_election.cc:288] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b5d1ed0b161417688f43a878b41ef3d (127.29.106.62:46573), e0ad6642bf744796b25dfefd04d3f245 (127.29.106.61:46755)
I20260811 22:26:09.253786 12047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:09.249088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:46573 (local address 127.0.0.1:42858)
0811 22:26:09.249608 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:09.253483 (+ 3875us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":559,"threads_started":1}
W20260811 22:26:09.254223 11012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.106.62:46573: connect: Connection refused (error 111)
I20260811 22:26:09.258330 12045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:09.248631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:46573 (local address 127.0.0.1:42844)
0811 22:26:09.257841 (+ 9210us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:09.258071 (+ 230us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":9094,"threads_started":1}
I20260811 22:26:09.261662 12049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:09.249454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.61:46755 (local address 127.0.0.1:56804)
0811 22:26:09.249961 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:09.249982 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:09.250722 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:09.253829 (+ 3107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:09.253834 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:09.253859 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:26:09.254236 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:09.254250 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:09.256278 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:09.256284 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:09.258803 (+ 2519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:09.258823 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:09.260828 (+ 2005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:09.260834 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:09.261037 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:09.261079 (+ 42us) client_negotiation.cc:769] Sending connection context
0811 22:26:09.261176 (+ 97us) client_negotiation.cc:240] Negotiation successful
0811 22:26:09.261310 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":333,"threads_started":1}
I20260811 22:26:09.262401 12050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:09.249864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56804 (local address 127.29.106.61:46755)
0811 22:26:09.253525 (+ 3661us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:09.253532 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:09.253560 (+ 28us) server_negotiation.cc:407] Connection header received
0811 22:26:09.253630 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:09.253636 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:09.253711 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:09.253805 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:26:09.254815 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:09.256151 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:09.259157 (+ 3006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:09.260525 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:09.260626 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:09.260729 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:09.261902 (+ 1173us) server_negotiation.cc:299] Negotiation successful
0811 22:26:09.262050 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3527,"thread_start_us":3318,"threads_started":1}
I20260811 22:26:09.263141 10669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "800b0066b8d34ef98b9c2bc24409ee79" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e0ad6642bf744796b25dfefd04d3f245" is_pre_election: true
I20260811 22:26:09.263458 10669 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 800b0066b8d34ef98b9c2bc24409ee79 in term 1.
I20260811 22:26:09.264073 11013 leader_election.cc:302] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 800b0066b8d34ef98b9c2bc24409ee79, e0ad6642bf744796b25dfefd04d3f245; no voters:
I20260811 22:26:09.264420 12037 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260811 22:26:09.264576 12037 raft_consensus.cc:482] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2b5d1ed0b161417688f43a878b41ef3d)
I20260811 22:26:09.264709 12037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [term 1 FOLLOWER]: Advancing to term 2
I20260811 22:26:09.265355 12049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:09.264953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:46573 (local address 127.0.0.1:42874)
0811 22:26:09.265118 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:09.265203 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260811 22:26:09.265590 11012 leader_election.cc:334] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b5d1ed0b161417688f43a878b41ef3d (127.29.106.62:46573): Network error: Client connection negotiation failed: client connection to 127.29.106.62:46573: connect: Connection refused (error 111)
W20260811 22:26:09.265406 10614 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.106.62:46573: connect: Connection refused (error 111)
I20260811 22:26:09.266856 12045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:09.248906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:40553 (local address 127.0.0.1:47158)
0811 22:26:09.259298 (+ 10392us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:09.259317 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:09.259454 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:09.260106 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:09.260111 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:09.260138 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:26:09.260451 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:09.260462 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:09.262353 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:09.262358 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:09.264282 (+ 1924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:09.264298 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:09.266192 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:09.266199 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:09.266345 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:09.266368 (+ 23us) client_negotiation.cc:769] Sending connection context
0811 22:26:09.266447 (+ 79us) client_negotiation.cc:240] Negotiation successful
0811 22:26:09.266577 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10279,"thread_start_us":9561,"threads_started":1}
I20260811 22:26:09.267006 12046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:09.248867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47158 (local address 127.29.106.60:40553)
0811 22:26:09.259038 (+ 10171us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:09.259044 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:09.259586 (+ 542us) server_negotiation.cc:407] Connection header received
0811 22:26:09.259650 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:09.259654 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:09.259793 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:09.259875 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:26:09.260647 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:09.262205 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:09.264475 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:09.265986 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:09.266112 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:09.266221 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:09.266506 (+ 285us) server_negotiation.cc:299] Negotiation successful
0811 22:26:09.266685 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10039,"thread_start_us":10062,"threads_started":1}
I20260811 22:26:09.267738 11061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e0ad6642bf744796b25dfefd04d3f245" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "800b0066b8d34ef98b9c2bc24409ee79" is_pre_election: true
I20260811 22:26:09.268231 12037 raft_consensus.cc:504] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: false } } peers { permanent_uuid: "800b0066b8d34ef98b9c2bc24409ee79" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 40553 } attrs { promote: false } }
I20260811 22:26:09.268741 12045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:09.268375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:46573 (local address 127.0.0.1:42878)
0811 22:26:09.268513 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:09.268596 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260811 22:26:09.268839 11061 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e0ad6642bf744796b25dfefd04d3f245 in current term 2: Already voted for candidate 800b0066b8d34ef98b9c2bc24409ee79 in this term.
W20260811 22:26:09.269137 10614 leader_election.cc:334] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2b5d1ed0b161417688f43a878b41ef3d (127.29.106.62:46573): Network error: Client connection negotiation failed: client connection to 127.29.106.62:46573: connect: Connection refused (error 111)
I20260811 22:26:09.269207 12037 leader_election.cc:288] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [CANDIDATE]: Term 2 election: Requested vote from peers 2b5d1ed0b161417688f43a878b41ef3d (127.29.106.62:46573), e0ad6642bf744796b25dfefd04d3f245 (127.29.106.61:46755)
I20260811 22:26:09.269362 12049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:09.268951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:46573 (local address 127.0.0.1:42890)
0811 22:26:09.269104 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:09.269206 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260811 22:26:09.269459 10609 leader_election.cc:302] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0ad6642bf744796b25dfefd04d3f245; no voters: 2b5d1ed0b161417688f43a878b41ef3d, 800b0066b8d34ef98b9c2bc24409ee79
I20260811 22:26:09.269637 10669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "800b0066b8d34ef98b9c2bc24409ee79" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e0ad6642bf744796b25dfefd04d3f245"
I20260811 22:26:09.269771 12031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245 [term 1 FOLLOWER]: Advancing to term 2
I20260811 22:26:09.270255 12049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:09.269913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:46573 (local address 127.0.0.1:42902)
0811 22:26:09.270057 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:09.270117 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260811 22:26:09.270416 11012 leader_election.cc:334] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 2b5d1ed0b161417688f43a878b41ef3d (127.29.106.62:46573): Network error: Client connection negotiation failed: client connection to 127.29.106.62:46573: connect: Connection refused (error 111)
I20260811 22:26:09.272002 12031 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 22:26:09.273960 10669 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 800b0066b8d34ef98b9c2bc24409ee79 in term 2.
I20260811 22:26:09.275175 11013 leader_election.cc:302] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 800b0066b8d34ef98b9c2bc24409ee79, e0ad6642bf744796b25dfefd04d3f245; no voters: 2b5d1ed0b161417688f43a878b41ef3d
I20260811 22:26:09.275506 12037 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [term 2 FOLLOWER]: Leader election won for term 2
I20260811 22:26:09.275882 12037 raft_consensus.cc:686] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [term 2 LEADER]: Becoming Leader. State: Replica: 800b0066b8d34ef98b9c2bc24409ee79, State: Running, Role: LEADER
I20260811 22:26:09.276237 12037 consensus_queue.cc:237] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: false } } peers { permanent_uuid: "800b0066b8d34ef98b9c2bc24409ee79" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 40553 } attrs { promote: false } }
I20260811 22:26:09.277861 12057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 800b0066b8d34ef98b9c2bc24409ee79. Latest consensus state: current_term: 2 leader_uuid: "800b0066b8d34ef98b9c2bc24409ee79" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: false } } peers { permanent_uuid: "800b0066b8d34ef98b9c2bc24409ee79" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 40553 } attrs { promote: false } } }
I20260811 22:26:09.278309 12057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [sys.catalog]: This master's current role is: LEADER
I20260811 22:26:09.279057 12059 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:26:09.284336 12059 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:26:09.285508 12059 catalog_manager.cc:1259] Loaded cluster ID: 8731d947535c4e8c8f3224f1af27f098
I20260811 22:26:09.285597 12059 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:26:09.287132 12059 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:26:09.288241 12059 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79: Loaded TSK: 0
I20260811 22:26:09.289399 12059 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:26:09.302894 11045 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47148:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:26:09.309679 11045 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:26:09.313694 12049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:09.313264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:46573 (local address 127.0.0.1:42930)
0811 22:26:09.313446 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:09.313525 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260811 22:26:09.313938 10669 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245 [term 2 FOLLOWER]: Refusing update from remote peer 800b0066b8d34ef98b9c2bc24409ee79: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260811 22:26:09.314679 12049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:09.314312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:46573 (local address 127.0.0.1:42936)
0811 22:26:09.314480 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:09.314564 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260811 22:26:09.314702 12057 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260811 22:26:09.315016 11012 consensus_peers.cc:487] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 -> Peer 2b5d1ed0b161417688f43a878b41ef3d (127.29.106.62:46573): Couldn't send request to peer 2b5d1ed0b161417688f43a878b41ef3d. Status: Network error: Client connection negotiation failed: client connection to 127.29.106.62:46573: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260811 22:26:09.317423 12031 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 800b0066b8d34ef98b9c2bc24409ee79. Latest consensus state: current_term: 2 leader_uuid: "800b0066b8d34ef98b9c2bc24409ee79" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: false } } peers { permanent_uuid: "800b0066b8d34ef98b9c2bc24409ee79" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 40553 } attrs { promote: false } } }
I20260811 22:26:09.317701 12031 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:26:09.318928 12057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "800b0066b8d34ef98b9c2bc24409ee79" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: false } } peers { permanent_uuid: "800b0066b8d34ef98b9c2bc24409ee79" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 40553 } attrs { promote: false } } }
I20260811 22:26:09.319294 12057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79 [sys.catalog]: This master's current role is: LEADER
I20260811 22:26:09.320109 12031 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "800b0066b8d34ef98b9c2bc24409ee79" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2b5d1ed0b161417688f43a878b41ef3d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 46573 } } peers { permanent_uuid: "e0ad6642bf744796b25dfefd04d3f245" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 46755 } attrs { promote: false } } peers { permanent_uuid: "800b0066b8d34ef98b9c2bc24409ee79" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 40553 } attrs { promote: false } } }
I20260811 22:26:09.320411 12031 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0ad6642bf744796b25dfefd04d3f245 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:26:09.341858 12049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:09.333334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:35211 (local address 127.0.0.1:40078)
0811 22:26:09.333595 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:09.333614 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:09.333762 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:09.335117 (+ 1355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:09.335123 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:09.335152 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:26:09.335419 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:09.335431 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:09.337187 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:09.337193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:09.339233 (+ 2040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:09.339250 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:09.341033 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:09.341040 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:09.341249 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:09.341290 (+ 41us) client_negotiation.cc:769] Sending connection context
0811 22:26:09.341357 (+ 67us) client_negotiation.cc:240] Negotiation successful
0811 22:26:09.341517 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142}
I20260811 22:26:09.343392 12075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:09.333468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40078 (local address 127.29.106.1:35211)
0811 22:26:09.334749 (+ 1281us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:09.334757 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:09.334779 (+ 22us) server_negotiation.cc:407] Connection header received
0811 22:26:09.334859 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:09.334865 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:09.334946 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:09.335576 (+ 630us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:26:09.335794 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:09.337029 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:09.339497 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:09.340833 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:09.342737 (+ 1904us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:09.342849 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:09.342905 (+ 56us) server_negotiation.cc:299] Negotiation successful
0811 22:26:09.343059 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1148,"thread_start_us":1024,"threads_started":1}
I20260811 22:26:09.344578 11242 tablet_service.cc:1467] Processing CreateTablet for tablet bc2e26d75d1a4409b66d9ec61de81595 (DEFAULT_TABLE table=test-table [id=182a16858e464260a76b487ebb1e315a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:26:09.345091 11242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc2e26d75d1a4409b66d9ec61de81595. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:26:09.350943 12049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:09.334228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.3:38811 (local address 127.0.0.1:42290)
0811 22:26:09.342105 (+ 7877us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:09.342124 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:09.342224 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:09.345270 (+ 3046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:09.345274 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:09.345299 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:26:09.345488 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:09.345499 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:09.347167 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:09.347171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:09.348746 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:09.348759 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:09.350137 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:09.350143 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:09.350385 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:09.350420 (+ 35us) client_negotiation.cc:769] Sending connection context
0811 22:26:09.350510 (+ 90us) client_negotiation.cc:240] Negotiation successful
0811 22:26:09.350650 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7783,"thread_start_us":10170,"threads_started":1}
I20260811 22:26:09.350968 12082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:09.334437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42290 (local address 127.29.106.3:38811)
0811 22:26:09.344930 (+ 10493us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:09.344937 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:09.344957 (+ 20us) server_negotiation.cc:407] Connection header received
0811 22:26:09.345027 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:09.345032 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:09.345107 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:09.345646 (+ 539us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:26:09.345838 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:09.347002 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:09.348933 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:09.349978 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:09.350120 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:09.350214 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:09.350555 (+ 341us) server_negotiation.cc:299] Negotiation successful
0811 22:26:09.350703 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10366,"thread_start_us":10269,"threads_started":1}
I20260811 22:26:09.351999 12085 tablet_bootstrap.cc:492] T bc2e26d75d1a4409b66d9ec61de81595 P f16cafe5e19446e782da1ed0fcd4a964: Bootstrap starting.
I20260811 22:26:09.352003 11786 tablet_service.cc:1467] Processing CreateTablet for tablet bc2e26d75d1a4409b66d9ec61de81595 (DEFAULT_TABLE table=test-table [id=182a16858e464260a76b487ebb1e315a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:26:09.352459 11786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc2e26d75d1a4409b66d9ec61de81595. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:26:09.353629 12085 tablet_bootstrap.cc:654] T bc2e26d75d1a4409b66d9ec61de81595 P f16cafe5e19446e782da1ed0fcd4a964: Neither blocks nor log segments found. Creating new log.
I20260811 22:26:09.353796 12047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:09.333821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.2:39773 (local address 127.0.0.1:54008)
0811 22:26:09.333959 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:09.333975 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:09.334063 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:09.347822 (+ 13759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:09.347827 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:09.347854 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:26:09.348166 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:09.348178 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:09.349694 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:09.349700 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:09.351603 (+ 1903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:09.351618 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:09.353108 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:09.353114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:09.353275 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:09.353308 (+ 33us) client_negotiation.cc:769] Sending connection context
0811 22:26:09.353407 (+ 99us) client_negotiation.cc:240] Negotiation successful
0811 22:26:09.353535 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260811 22:26:09.353853 12076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:09.334182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54008 (local address 127.29.106.2:39773)
0811 22:26:09.347494 (+ 13312us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:09.347500 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:09.347520 (+ 20us) server_negotiation.cc:407] Connection header received
0811 22:26:09.347602 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:09.347607 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:09.347680 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:09.347780 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:26:09.348468 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:09.349537 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:09.351788 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:09.352908 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:09.353051 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:09.353157 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:09.353433 (+ 276us) server_negotiation.cc:299] Negotiation successful
0811 22:26:09.353583 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13186,"thread_start_us":13810,"threads_started":1}
I20260811 22:26:09.354836 11509 tablet_service.cc:1467] Processing CreateTablet for tablet bc2e26d75d1a4409b66d9ec61de81595 (DEFAULT_TABLE table=test-table [id=182a16858e464260a76b487ebb1e315a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:26:09.355291 11509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc2e26d75d1a4409b66d9ec61de81595. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:26:09.358812 12088 tablet_bootstrap.cc:492] T bc2e26d75d1a4409b66d9ec61de81595 P 5828483962134e1b972872a8dd583692: Bootstrap starting.
I20260811 22:26:09.360775 12088 tablet_bootstrap.cc:654] T bc2e26d75d1a4409b66d9ec61de81595 P 5828483962134e1b972872a8dd583692: Neither blocks nor log segments found. Creating new log.
I20260811 22:26:09.361991 12089 tablet_bootstrap.cc:492] T bc2e26d75d1a4409b66d9ec61de81595 P 7ff22bba56ee42879d1d18e2630d0fc3: Bootstrap starting.
I20260811 22:26:09.363463 12085 tablet_bootstrap.cc:492] T bc2e26d75d1a4409b66d9ec61de81595 P f16cafe5e19446e782da1ed0fcd4a964: No bootstrap required, opened a new log
I20260811 22:26:09.363600 12085 ts_tablet_manager.cc:1397] T bc2e26d75d1a4409b66d9ec61de81595 P f16cafe5e19446e782da1ed0fcd4a964: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.005s
I20260811 22:26:09.363616 12088 tablet_bootstrap.cc:492] T bc2e26d75d1a4409b66d9ec61de81595 P 5828483962134e1b972872a8dd583692: No bootstrap required, opened a new log
I20260811 22:26:09.363745 12088 ts_tablet_manager.cc:1397] T bc2e26d75d1a4409b66d9ec61de81595 P 5828483962134e1b972872a8dd583692: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260811 22:26:09.363780 12089 tablet_bootstrap.cc:654] T bc2e26d75d1a4409b66d9ec61de81595 P 7ff22bba56ee42879d1d18e2630d0fc3: Neither blocks nor log segments found. Creating new log.
I20260811 22:26:09.364285 12085 raft_consensus.cc:348] T bc2e26d75d1a4409b66d9ec61de81595 P f16cafe5e19446e782da1ed0fcd4a964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f16cafe5e19446e782da1ed0fcd4a964" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 35211 } } peers { permanent_uuid: "7ff22bba56ee42879d1d18e2630d0fc3" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39773 } } peers { permanent_uuid: "5828483962134e1b972872a8dd583692" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38811 } }
I20260811 22:26:09.364470 12085 raft_consensus.cc:374] T bc2e26d75d1a4409b66d9ec61de81595 P f16cafe5e19446e782da1ed0fcd4a964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:26:09.364578 12085 raft_consensus.cc:729] T bc2e26d75d1a4409b66d9ec61de81595 P f16cafe5e19446e782da1ed0fcd4a964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f16cafe5e19446e782da1ed0fcd4a964, State: Initialized, Role: FOLLOWER
I20260811 22:26:09.364568 12088 raft_consensus.cc:348] T bc2e26d75d1a4409b66d9ec61de81595 P 5828483962134e1b972872a8dd583692 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f16cafe5e19446e782da1ed0fcd4a964" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 35211 } } peers { permanent_uuid: "7ff22bba56ee42879d1d18e2630d0fc3" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39773 } } peers { permanent_uuid: "5828483962134e1b972872a8dd583692" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38811 } }
I20260811 22:26:09.364822 12088 raft_consensus.cc:374] T bc2e26d75d1a4409b66d9ec61de81595 P 5828483962134e1b972872a8dd583692 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:26:09.364914 12088 raft_consensus.cc:729] T bc2e26d75d1a4409b66d9ec61de81595 P 5828483962134e1b972872a8dd583692 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5828483962134e1b972872a8dd583692, State: Initialized, Role: FOLLOWER
I20260811 22:26:09.364887 12085 consensus_queue.cc:260] T bc2e26d75d1a4409b66d9ec61de81595 P f16cafe5e19446e782da1ed0fcd4a964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f16cafe5e19446e782da1ed0fcd4a964" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 35211 } } peers { permanent_uuid: "7ff22bba56ee42879d1d18e2630d0fc3" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39773 } } peers { permanent_uuid: "5828483962134e1b972872a8dd583692" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38811 } }
I20260811 22:26:09.365151 12088 consensus_queue.cc:260] T bc2e26d75d1a4409b66d9ec61de81595 P 5828483962134e1b972872a8dd583692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f16cafe5e19446e782da1ed0fcd4a964" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 35211 } } peers { permanent_uuid: "7ff22bba56ee42879d1d18e2630d0fc3" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39773 } } peers { permanent_uuid: "5828483962134e1b972872a8dd583692" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38811 } }
I20260811 22:26:09.365922 12085 ts_tablet_manager.cc:1428] T bc2e26d75d1a4409b66d9ec61de81595 P f16cafe5e19446e782da1ed0fcd4a964: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260811 22:26:09.365964 12089 tablet_bootstrap.cc:492] T bc2e26d75d1a4409b66d9ec61de81595 P 7ff22bba56ee42879d1d18e2630d0fc3: No bootstrap required, opened a new log
I20260811 22:26:09.366182 12089 ts_tablet_manager.cc:1397] T bc2e26d75d1a4409b66d9ec61de81595 P 7ff22bba56ee42879d1d18e2630d0fc3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260811 22:26:09.367010 12089 raft_consensus.cc:348] T bc2e26d75d1a4409b66d9ec61de81595 P 7ff22bba56ee42879d1d18e2630d0fc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f16cafe5e19446e782da1ed0fcd4a964" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 35211 } } peers { permanent_uuid: "7ff22bba56ee42879d1d18e2630d0fc3" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39773 } } peers { permanent_uuid: "5828483962134e1b972872a8dd583692" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38811 } }
I20260811 22:26:09.367336 12089 raft_consensus.cc:374] T bc2e26d75d1a4409b66d9ec61de81595 P 7ff22bba56ee42879d1d18e2630d0fc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:26:09.367475 12089 raft_consensus.cc:729] T bc2e26d75d1a4409b66d9ec61de81595 P 7ff22bba56ee42879d1d18e2630d0fc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ff22bba56ee42879d1d18e2630d0fc3, State: Initialized, Role: FOLLOWER
I20260811 22:26:09.367800 12089 consensus_queue.cc:260] T bc2e26d75d1a4409b66d9ec61de81595 P 7ff22bba56ee42879d1d18e2630d0fc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f16cafe5e19446e782da1ed0fcd4a964" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 35211 } } peers { permanent_uuid: "7ff22bba56ee42879d1d18e2630d0fc3" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39773 } } peers { permanent_uuid: "5828483962134e1b972872a8dd583692" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38811 } }
I20260811 22:26:09.369277 11371 heartbeater.cc:503] Master 127.29.106.60:40553 was elected leader, sending a full tablet report...
I20260811 22:26:09.375893 12088 ts_tablet_manager.cc:1428] T bc2e26d75d1a4409b66d9ec61de81595 P 5828483962134e1b972872a8dd583692: Time spent starting tablet: real 0.012s user 0.008s sys 0.003s
I20260811 22:26:09.377750 11903 heartbeater.cc:503] Master 127.29.106.60:40553 was elected leader, sending a full tablet report...
I20260811 22:26:09.379375 12096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:09.366530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:46573 (local address 127.29.106.1:58179)
0811 22:26:09.378890 (+ 12360us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:09.379063 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12217,"threads_started":1}
I20260811 22:26:09.380436 12089 ts_tablet_manager.cc:1428] T bc2e26d75d1a4409b66d9ec61de81595 P 7ff22bba56ee42879d1d18e2630d0fc3: Time spent starting tablet: real 0.014s user 0.002s sys 0.011s
I20260811 22:26:09.380863 12096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:09.380259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:46573 (local address 127.29.106.1:53941)
0811 22:26:09.380639 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:09.380737 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280}
I20260811 22:26:09.382349 11640 heartbeater.cc:503] Master 127.29.106.60:40553 was elected leader, sending a full tablet report...
I20260811 22:26:09.383101 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:09.381149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:46573 (local address 127.29.106.2:54065)
0811 22:26:09.382679 (+ 1530us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:09.382855 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1394,"threads_started":1}
I20260811 22:26:09.384443 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:09.377036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:46573 (local address 127.29.106.3:40805)
0811 22:26:09.384054 (+ 7018us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:09.384227 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6872,"threads_started":1}
I20260811 22:26:09.384613 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:09.384255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:46573 (local address 127.29.106.2:34939)
0811 22:26:09.384394 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:09.384490 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260811 22:26:09.385499 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:09.385216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:46573 (local address 127.29.106.3:60905)
0811 22:26:09.385325 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:09.385392 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260811 22:26:09.472242 12099 raft_consensus.cc:482] T bc2e26d75d1a4409b66d9ec61de81595 P 7ff22bba56ee42879d1d18e2630d0fc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:26:09.472499 12099 raft_consensus.cc:504] T bc2e26d75d1a4409b66d9ec61de81595 P 7ff22bba56ee42879d1d18e2630d0fc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f16cafe5e19446e782da1ed0fcd4a964" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 35211 } } peers { permanent_uuid: "7ff22bba56ee42879d1d18e2630d0fc3" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39773 } } peers { permanent_uuid: "5828483962134e1b972872a8dd583692" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38811 } }
I20260811 22:26:09.473847 12095 raft_consensus.cc:482] T bc2e26d75d1a4409b66d9ec61de81595 P 5828483962134e1b972872a8dd583692 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:26:09.474174 12099 leader_election.cc:288] T bc2e26d75d1a4409b66d9ec61de81595 P 7ff22bba56ee42879d1d18e2630d0fc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f16cafe5e19446e782da1ed0fcd4a964 (127.29.106.1:35211), 5828483962134e1b972872a8dd583692 (127.29.106.3:38811)
I20260811 22:26:09.474056 12095 raft_consensus.cc:504] T bc2e26d75d1a4409b66d9ec61de81595 P 5828483962134e1b972872a8dd583692 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f16cafe5e19446e782da1ed0fcd4a964" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 35211 } } peers { permanent_uuid: "7ff22bba56ee42879d1d18e2630d0fc3" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39773 } } peers { permanent_uuid: "5828483962134e1b972872a8dd583692" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38811 } }
I20260811 22:26:09.476218 12095 leader_election.cc:288] T bc2e26d75d1a4409b66d9ec61de81595 P 5828483962134e1b972872a8dd583692 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f16cafe5e19446e782da1ed0fcd4a964 (127.29.106.1:35211), 7ff22bba56ee42879d1d18e2630d0fc3 (127.29.106.2:39773)
I20260811 22:26:09.482051 12075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:09.474693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:38759 (local address 127.29.106.1:35211)
0811 22:26:09.474897 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:09.474903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:09.474924 (+ 21us) server_negotiation.cc:407] Connection header received
0811 22:26:09.475003 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:09.475008 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:09.475095 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:09.475190 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:26:09.475822 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:09.476874 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:09.479982 (+ 3108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:09.480975 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:09.481087 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:09.481154 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:09.481675 (+ 521us) server_negotiation.cc:299] Negotiation successful
0811 22:26:09.481800 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260811 22:26:09.482805 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:09.474354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:35211 (local address 127.29.106.2:38759)
0811 22:26:09.474624 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:09.474644 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:09.474768 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:09.475210 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:09.475215 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:09.475290 (+ 75us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:26:09.475653 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:09.475667 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:09.477440 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:09.477445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:09.479806 (+ 2361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:09.479821 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:09.481328 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:09.481333 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:09.481485 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:09.481514 (+ 29us) client_negotiation.cc:769] Sending connection context
0811 22:26:09.482278 (+ 764us) client_negotiation.cc:240] Negotiation successful
0811 22:26:09.482495 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":45}
I20260811 22:26:09.483598 11272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc2e26d75d1a4409b66d9ec61de81595" candidate_uuid: "7ff22bba56ee42879d1d18e2630d0fc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f16cafe5e19446e782da1ed0fcd4a964" is_pre_election: true
I20260811 22:26:09.484138 11272 raft_consensus.cc:2454] T bc2e26d75d1a4409b66d9ec61de81595 P f16cafe5e19446e782da1ed0fcd4a964 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ff22bba56ee42879d1d18e2630d0fc3 in term 0.
I20260811 22:26:09.484530 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:09.476740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.2:39773 (local address 127.29.106.3:40525)
0811 22:26:09.477748 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:09.477763 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:09.477855 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:09.478243 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:09.478247 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:09.478284 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:26:09.478520 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:09.478533 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:09.479921 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:09.479926 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:09.481916 (+ 1990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:09.481931 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:09.483759 (+ 1828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:09.483766 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:09.483954 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:09.483987 (+ 33us) client_negotiation.cc:769] Sending connection context
0811 22:26:09.484085 (+ 98us) client_negotiation.cc:240] Negotiation successful
0811 22:26:09.484230 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":917,"thread_start_us":510,"threads_started":1}
I20260811 22:26:09.484582 12076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:09.477060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:40525 (local address 127.29.106.2:39773)
0811 22:26:09.477240 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:09.477246 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:09.477822 (+ 576us) server_negotiation.cc:407] Connection header received
0811 22:26:09.477949 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:09.477954 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:09.478022 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:09.478103 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:26:09.478682 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:09.479754 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:09.482103 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:09.483573 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:09.483725 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:09.483796 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:09.484112 (+ 316us) server_negotiation.cc:299] Negotiation successful
0811 22:26:09.484241 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":130,"server-negotiator.queue_time_us":51}
I20260811 22:26:09.485025 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:09.476385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:35211 (local address 127.29.106.3:60963)
0811 22:26:09.477389 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:09.477418 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:09.477542 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:09.478641 (+ 1099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:09.478645 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:09.478668 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:26:09.478986 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:09.478997 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:09.480680 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:09.480685 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:09.482642 (+ 1957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:09.482657 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:09.484302 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:09.484309 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:09.484494 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:09.484529 (+ 35us) client_negotiation.cc:769] Sending connection context
0811 22:26:09.484597 (+ 68us) client_negotiation.cc:240] Negotiation successful
0811 22:26:09.484719 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":872}
I20260811 22:26:09.485047 11425 leader_election.cc:302] T bc2e26d75d1a4409b66d9ec61de81595 P 7ff22bba56ee42879d1d18e2630d0fc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ff22bba56ee42879d1d18e2630d0fc3, f16cafe5e19446e782da1ed0fcd4a964; no voters:
I20260811 22:26:09.485466 12099 raft_consensus.cc:2781] T bc2e26d75d1a4409b66d9ec61de81595 P 7ff22bba56ee42879d1d18e2630d0fc3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:26:09.485594 12099 raft_consensus.cc:482] T bc2e26d75d1a4409b66d9ec61de81595 P 7ff22bba56ee42879d1d18e2630d0fc3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:26:09.485697 12099 raft_consensus.cc:3037] T bc2e26d75d1a4409b66d9ec61de81595 P 7ff22bba56ee42879d1d18e2630d0fc3 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:26:09.485651 11548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc2e26d75d1a4409b66d9ec61de81595" candidate_uuid: "5828483962134e1b972872a8dd583692" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ff22bba56ee42879d1d18e2630d0fc3" is_pre_election: true
I20260811 22:26:09.485852 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:09.476604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:60963 (local address 127.29.106.1:35211)
0811 22:26:09.478351 (+ 1747us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:09.478355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:09.478375 (+ 20us) server_negotiation.cc:407] Connection header received
0811 22:26:09.478437 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:09.478442 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:09.478503 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:09.479134 (+ 631us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:26:09.479370 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:09.480517 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:09.482806 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:09.484064 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:09.485307 (+ 1243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:09.485373 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:09.485429 (+ 56us) server_negotiation.cc:299] Negotiation successful
0811 22:26:09.485547 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1633,"thread_start_us":1486,"threads_started":1}
I20260811 22:26:09.486829 12082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:09.474540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:53331 (local address 127.29.106.3:38811)
0811 22:26:09.474721 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:09.474727 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:09.475368 (+ 641us) server_negotiation.cc:407] Connection header received
0811 22:26:09.475648 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:09.475654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:09.475730 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:09.475816 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:26:09.476463 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:09.477522 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:09.484777 (+ 7255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:09.485957 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:09.486103 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:09.486168 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:09.486489 (+ 321us) server_negotiation.cc:299] Negotiation successful
0811 22:26:09.486610 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260811 22:26:09.486840 12104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:09.474403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.3:38811 (local address 127.29.106.2:53331)
0811 22:26:09.475256 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:09.475295 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:09.475495 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:09.475932 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:09.475935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:09.475954 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:26:09.476262 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:09.476276 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:09.477696 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:09.477701 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:09.484600 (+ 6899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:09.484616 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:09.486152 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:09.486156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:09.486323 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:09.486354 (+ 31us) client_negotiation.cc:769] Sending connection context
0811 22:26:09.486453 (+ 99us) client_negotiation.cc:240] Negotiation successful
0811 22:26:09.486543 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"mutex_wait_us":1519,"thread_start_us":579,"threads_started":1}
I20260811 22:26:09.487776 11272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc2e26d75d1a4409b66d9ec61de81595" candidate_uuid: "5828483962134e1b972872a8dd583692" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f16cafe5e19446e782da1ed0fcd4a964" is_pre_election: true
I20260811 22:26:09.487776 11821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc2e26d75d1a4409b66d9ec61de81595" candidate_uuid: "7ff22bba56ee42879d1d18e2630d0fc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5828483962134e1b972872a8dd583692" is_pre_election: true
I20260811 22:26:09.487998 11272 raft_consensus.cc:2454] T bc2e26d75d1a4409b66d9ec61de81595 P f16cafe5e19446e782da1ed0fcd4a964 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5828483962134e1b972872a8dd583692 in term 0.
I20260811 22:26:09.488003 11821 raft_consensus.cc:2454] T bc2e26d75d1a4409b66d9ec61de81595 P 5828483962134e1b972872a8dd583692 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ff22bba56ee42879d1d18e2630d0fc3 in term 0.
I20260811 22:26:09.488427 12099 raft_consensus.cc:504] T bc2e26d75d1a4409b66d9ec61de81595 P 7ff22bba56ee42879d1d18e2630d0fc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f16cafe5e19446e782da1ed0fcd4a964" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 35211 } } peers { permanent_uuid: "7ff22bba56ee42879d1d18e2630d0fc3" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39773 } } peers { permanent_uuid: "5828483962134e1b972872a8dd583692" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38811 } }
I20260811 22:26:09.488471 11692 leader_election.cc:302] T bc2e26d75d1a4409b66d9ec61de81595 P 5828483962134e1b972872a8dd583692 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5828483962134e1b972872a8dd583692, f16cafe5e19446e782da1ed0fcd4a964; no voters:
I20260811 22:26:09.488807 11548 raft_consensus.cc:2379] T bc2e26d75d1a4409b66d9ec61de81595 P 7ff22bba56ee42879d1d18e2630d0fc3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5828483962134e1b972872a8dd583692 in current term 1: Already voted for candidate 7ff22bba56ee42879d1d18e2630d0fc3 in this term.
I20260811 22:26:09.488814 12095 raft_consensus.cc:2781] T bc2e26d75d1a4409b66d9ec61de81595 P 5828483962134e1b972872a8dd583692 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:26:09.488934 12095 raft_consensus.cc:482] T bc2e26d75d1a4409b66d9ec61de81595 P 5828483962134e1b972872a8dd583692 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:26:09.489032 12095 raft_consensus.cc:3037] T bc2e26d75d1a4409b66d9ec61de81595 P 5828483962134e1b972872a8dd583692 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:26:09.489130 12099 leader_election.cc:288] T bc2e26d75d1a4409b66d9ec61de81595 P 7ff22bba56ee42879d1d18e2630d0fc3 [CANDIDATE]: Term 1 election: Requested vote from peers f16cafe5e19446e782da1ed0fcd4a964 (127.29.106.1:35211), 5828483962134e1b972872a8dd583692 (127.29.106.3:38811)
I20260811 22:26:09.489627 11821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc2e26d75d1a4409b66d9ec61de81595" candidate_uuid: "7ff22bba56ee42879d1d18e2630d0fc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5828483962134e1b972872a8dd583692"
I20260811 22:26:09.489759 11272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc2e26d75d1a4409b66d9ec61de81595" candidate_uuid: "7ff22bba56ee42879d1d18e2630d0fc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f16cafe5e19446e782da1ed0fcd4a964"
I20260811 22:26:09.489933 11272 raft_consensus.cc:3037] T bc2e26d75d1a4409b66d9ec61de81595 P f16cafe5e19446e782da1ed0fcd4a964 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:26:09.492707 11272 raft_consensus.cc:2454] T bc2e26d75d1a4409b66d9ec61de81595 P f16cafe5e19446e782da1ed0fcd4a964 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ff22bba56ee42879d1d18e2630d0fc3 in term 1.
I20260811 22:26:09.493122 11425 leader_election.cc:302] T bc2e26d75d1a4409b66d9ec61de81595 P 7ff22bba56ee42879d1d18e2630d0fc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ff22bba56ee42879d1d18e2630d0fc3, f16cafe5e19446e782da1ed0fcd4a964; no voters:
I20260811 22:26:09.493471 12099 raft_consensus.cc:2781] T bc2e26d75d1a4409b66d9ec61de81595 P 7ff22bba56ee42879d1d18e2630d0fc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:26:09.494952 12099 raft_consensus.cc:686] T bc2e26d75d1a4409b66d9ec61de81595 P 7ff22bba56ee42879d1d18e2630d0fc3 [term 1 LEADER]: Becoming Leader. State: Replica: 7ff22bba56ee42879d1d18e2630d0fc3, State: Running, Role: LEADER
I20260811 22:26:09.495306 12095 raft_consensus.cc:504] T bc2e26d75d1a4409b66d9ec61de81595 P 5828483962134e1b972872a8dd583692 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f16cafe5e19446e782da1ed0fcd4a964" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 35211 } } peers { permanent_uuid: "7ff22bba56ee42879d1d18e2630d0fc3" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39773 } } peers { permanent_uuid: "5828483962134e1b972872a8dd583692" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38811 } }
I20260811 22:26:09.495690 11821 raft_consensus.cc:2379] T bc2e26d75d1a4409b66d9ec61de81595 P 5828483962134e1b972872a8dd583692 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7ff22bba56ee42879d1d18e2630d0fc3 in current term 1: Already voted for candidate 5828483962134e1b972872a8dd583692 in this term.
I20260811 22:26:09.495801 12099 consensus_queue.cc:237] T bc2e26d75d1a4409b66d9ec61de81595 P 7ff22bba56ee42879d1d18e2630d0fc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f16cafe5e19446e782da1ed0fcd4a964" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 35211 } } peers { permanent_uuid: "7ff22bba56ee42879d1d18e2630d0fc3" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39773 } } peers { permanent_uuid: "5828483962134e1b972872a8dd583692" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38811 } }
I20260811 22:26:09.497373 12095 leader_election.cc:288] T bc2e26d75d1a4409b66d9ec61de81595 P 5828483962134e1b972872a8dd583692 [CANDIDATE]: Term 1 election: Requested vote from peers f16cafe5e19446e782da1ed0fcd4a964 (127.29.106.1:35211), 7ff22bba56ee42879d1d18e2630d0fc3 (127.29.106.2:39773)
I20260811 22:26:09.497506 11272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc2e26d75d1a4409b66d9ec61de81595" candidate_uuid: "5828483962134e1b972872a8dd583692" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f16cafe5e19446e782da1ed0fcd4a964"
I20260811 22:26:09.497793 11272 raft_consensus.cc:2379] T bc2e26d75d1a4409b66d9ec61de81595 P f16cafe5e19446e782da1ed0fcd4a964 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5828483962134e1b972872a8dd583692 in current term 1: Already voted for candidate 7ff22bba56ee42879d1d18e2630d0fc3 in this term.
I20260811 22:26:09.498036 11548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc2e26d75d1a4409b66d9ec61de81595" candidate_uuid: "5828483962134e1b972872a8dd583692" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ff22bba56ee42879d1d18e2630d0fc3"
I20260811 22:26:09.498159 12104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:09.496386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:46573 (local address 127.29.106.2:32805)
0811 22:26:09.497859 (+ 1473us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:09.497991 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260811 22:26:09.499056 11692 leader_election.cc:302] T bc2e26d75d1a4409b66d9ec61de81595 P 5828483962134e1b972872a8dd583692 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5828483962134e1b972872a8dd583692; no voters: 7ff22bba56ee42879d1d18e2630d0fc3, f16cafe5e19446e782da1ed0fcd4a964
I20260811 22:26:09.499639 12095 raft_consensus.cc:2726] T bc2e26d75d1a4409b66d9ec61de81595 P 5828483962134e1b972872a8dd583692 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260811 22:26:09.502568 11046 catalog_manager.cc:5495] T bc2e26d75d1a4409b66d9ec61de81595 P 7ff22bba56ee42879d1d18e2630d0fc3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ff22bba56ee42879d1d18e2630d0fc3 (127.29.106.2). New cstate: current_term: 1 leader_uuid: "7ff22bba56ee42879d1d18e2630d0fc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f16cafe5e19446e782da1ed0fcd4a964" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 35211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ff22bba56ee42879d1d18e2630d0fc3" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5828483962134e1b972872a8dd583692" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38811 } health_report { overall_health: UNKNOWN } } }
I20260811 22:26:09.550426 12076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:09.545647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54012 (local address 127.29.106.2:39773)
0811 22:26:09.545794 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:09.545798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:09.545815 (+ 17us) server_negotiation.cc:407] Connection header received
0811 22:26:09.545986 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:09.545990 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:09.546068 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:09.546192 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:26:09.546468 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:09.547242 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:09.548507 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:09.548914 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:09.549004 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:09.549377 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:26:09.549934 (+ 557us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:26:09.550043 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:09.550130 (+ 87us) server_negotiation.cc:299] Negotiation successful
0811 22:26:09.550195 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 22:26:09.554404 11821 raft_consensus.cc:1261] T bc2e26d75d1a4409b66d9ec61de81595 P 5828483962134e1b972872a8dd583692 [term 1 FOLLOWER]: Refusing update from remote peer 7ff22bba56ee42879d1d18e2630d0fc3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:26:09.554688 11272 raft_consensus.cc:1261] T bc2e26d75d1a4409b66d9ec61de81595 P f16cafe5e19446e782da1ed0fcd4a964 [term 1 FOLLOWER]: Refusing update from remote peer 7ff22bba56ee42879d1d18e2630d0fc3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:26:09.555092 12108 consensus_queue.cc:1035] T bc2e26d75d1a4409b66d9ec61de81595 P 7ff22bba56ee42879d1d18e2630d0fc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5828483962134e1b972872a8dd583692" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:26:09.555569 12099 consensus_queue.cc:1035] T bc2e26d75d1a4409b66d9ec61de81595 P 7ff22bba56ee42879d1d18e2630d0fc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f16cafe5e19446e782da1ed0fcd4a964" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 35211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:26:09.558125 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:09.557689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:46573 (local address 127.29.106.3:40663)
0811 22:26:09.557835 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:09.557954 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260811 22:26:09.558620 12096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:09.557868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:46573 (local address 127.29.106.1:49381)
0811 22:26:09.558336 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:09.558461 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360}
I20260811 22:26:09.565910 12096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:09.565400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:46573 (local address 127.29.106.1:58097)
0811 22:26:09.565661 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:09.565747 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260811 22:26:09.566469 12104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:09.564209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:46573 (local address 127.29.106.2:48149)
0811 22:26:09.565418 (+ 1209us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:09.565508 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1114}
I20260811 22:26:09.567162 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:09.566569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:46573 (local address 127.29.106.3:34789)
0811 22:26:09.566885 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:09.566989 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196}
I20260811 22:26:09.567559 12104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:09.567097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:46573 (local address 127.29.106.2:46779)
0811 22:26:09.567315 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:09.567408 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260811 22:26:09.580937 11045 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:26:09.581183 11045 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:26:09.586161 11045 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 800b0066b8d34ef98b9c2bc24409ee79: Sending DeleteTablet for 3 replicas of tablet bc2e26d75d1a4409b66d9ec61de81595
I20260811 22:26:09.587245 11242 tablet_service.cc:1514] Processing DeleteTablet for tablet bc2e26d75d1a4409b66d9ec61de81595 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:26:09 UTC) from {username='oryx', principal='oryx/127.29.106.60@KRBTEST.COM'} at 127.0.0.1:40078
I20260811 22:26:09.587327 11786 tablet_service.cc:1514] Processing DeleteTablet for tablet bc2e26d75d1a4409b66d9ec61de81595 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:26:09 UTC) from {username='oryx', principal='oryx/127.29.106.60@KRBTEST.COM'} at 127.0.0.1:42290
I20260811 22:26:09.587353 11509 tablet_service.cc:1514] Processing DeleteTablet for tablet bc2e26d75d1a4409b66d9ec61de81595 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:26:09 UTC) from {username='oryx', principal='oryx/127.29.106.60@KRBTEST.COM'} at 127.0.0.1:54008
I20260811 22:26:09.589032 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 10963
I20260811 22:26:09.606418 12132 tablet_replica.cc:330] T bc2e26d75d1a4409b66d9ec61de81595 P f16cafe5e19446e782da1ed0fcd4a964: stopping tablet replica
I20260811 22:26:09.606645 12132 raft_consensus.cc:2229] T bc2e26d75d1a4409b66d9ec61de81595 P f16cafe5e19446e782da1ed0fcd4a964 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:26:09.606964 12132 raft_consensus.cc:2258] T bc2e26d75d1a4409b66d9ec61de81595 P f16cafe5e19446e782da1ed0fcd4a964 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:26:09.607642 12133 tablet_replica.cc:330] T bc2e26d75d1a4409b66d9ec61de81595 P 5828483962134e1b972872a8dd583692: stopping tablet replica
I20260811 22:26:09.607832 12133 raft_consensus.cc:2229] T bc2e26d75d1a4409b66d9ec61de81595 P 5828483962134e1b972872a8dd583692 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:26:09.607890 12135 tablet_replica.cc:330] T bc2e26d75d1a4409b66d9ec61de81595 P 7ff22bba56ee42879d1d18e2630d0fc3: stopping tablet replica
I20260811 22:26:09.608064 12135 raft_consensus.cc:2229] T bc2e26d75d1a4409b66d9ec61de81595 P 7ff22bba56ee42879d1d18e2630d0fc3 [term 1 LEADER]: Raft consensus shutting down.
I20260811 22:26:09.608146 12133 raft_consensus.cc:2258] T bc2e26d75d1a4409b66d9ec61de81595 P 5828483962134e1b972872a8dd583692 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:26:09.608984 12135 raft_consensus.cc:2258] T bc2e26d75d1a4409b66d9ec61de81595 P 7ff22bba56ee42879d1d18e2630d0fc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:26:09.609687 12132 ts_tablet_manager.cc:1905] T bc2e26d75d1a4409b66d9ec61de81595 P f16cafe5e19446e782da1ed0fcd4a964: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:26:09.610332 12135 ts_tablet_manager.cc:1905] T bc2e26d75d1a4409b66d9ec61de81595 P 7ff22bba56ee42879d1d18e2630d0fc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:26:09.611848 12133 ts_tablet_manager.cc:1905] T bc2e26d75d1a4409b66d9ec61de81595 P 5828483962134e1b972872a8dd583692: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:26:09.616214 12132 ts_tablet_manager.cc:1918] T bc2e26d75d1a4409b66d9ec61de81595 P f16cafe5e19446e782da1ed0fcd4a964: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:26:09.616357 12132 log.cc:1196] T bc2e26d75d1a4409b66d9ec61de81595 P f16cafe5e19446e782da1ed0fcd4a964: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-0/wal/wals/bc2e26d75d1a4409b66d9ec61de81595
I20260811 22:26:09.616833 12132 ts_tablet_manager.cc:1939] T bc2e26d75d1a4409b66d9ec61de81595 P f16cafe5e19446e782da1ed0fcd4a964: Deleting consensus metadata
I20260811 22:26:09.617267 12135 ts_tablet_manager.cc:1918] T bc2e26d75d1a4409b66d9ec61de81595 P 7ff22bba56ee42879d1d18e2630d0fc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:26:09.617481 12135 log.cc:1196] T bc2e26d75d1a4409b66d9ec61de81595 P 7ff22bba56ee42879d1d18e2630d0fc3: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-1/wal/wals/bc2e26d75d1a4409b66d9ec61de81595
I20260811 22:26:09.618100 12135 ts_tablet_manager.cc:1939] T bc2e26d75d1a4409b66d9ec61de81595 P 7ff22bba56ee42879d1d18e2630d0fc3: Deleting consensus metadata
I20260811 22:26:09.618074 11013 catalog_manager.cc:4841] TS f16cafe5e19446e782da1ed0fcd4a964 (127.29.106.1:35211): tablet bc2e26d75d1a4409b66d9ec61de81595 (table test-table [id=182a16858e464260a76b487ebb1e315a]) successfully deleted
I20260811 22:26:09.619495 11013 catalog_manager.cc:4841] TS 7ff22bba56ee42879d1d18e2630d0fc3 (127.29.106.2:39773): tablet bc2e26d75d1a4409b66d9ec61de81595 (table test-table [id=182a16858e464260a76b487ebb1e315a]) successfully deleted
I20260811 22:26:09.622500 12133 ts_tablet_manager.cc:1918] T bc2e26d75d1a4409b66d9ec61de81595 P 5828483962134e1b972872a8dd583692: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:26:09.622641 12133 log.cc:1196] T bc2e26d75d1a4409b66d9ec61de81595 P 5828483962134e1b972872a8dd583692: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786487105757681-30120-0/minicluster-data/ts-2/wal/wals/bc2e26d75d1a4409b66d9ec61de81595
I20260811 22:26:09.623111 12133 ts_tablet_manager.cc:1939] T bc2e26d75d1a4409b66d9ec61de81595 P 5828483962134e1b972872a8dd583692: Deleting consensus metadata
I20260811 22:26:09.623250 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:09.590112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40086 (local address 127.29.106.1:35211)
0811 22:26:09.590293 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:09.590299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:09.601078 (+ 10779us) server_negotiation.cc:407] Connection header received
0811 22:26:09.601923 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:09.601929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:09.602027 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:09.602113 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:09.608487 (+ 6374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:09.609638 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:09.611587 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:09.612212 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:09.612307 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:09.615889 (+ 3582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:09.615907 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:09.615926 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:09.615976 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:09.620037 (+ 4061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:09.620921 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:09.620932 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:09.620936 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:09.621117 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:09.621529 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:09.621534 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:09.621537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:09.622048 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:09.622347 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:09.622662 (+ 315us) server_negotiation.cc:299] Negotiation successful
0811 22:26:09.622902 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 22:26:09.623914 11362 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:26:09.624152 11007 catalog_manager.cc:4841] TS 5828483962134e1b972872a8dd583692 (127.29.106.3:38811): tablet bc2e26d75d1a4409b66d9ec61de81595 (table test-table [id=182a16858e464260a76b487ebb1e315a]) successfully deleted
I20260811 22:26:09.706852 12047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:09.706363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:46573 (local address 127.0.0.1:39404)
0811 22:26:09.706552 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:09.706662 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260811 22:26:09.708053 12047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:09.707675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:46573 (local address 127.0.0.1:39420)
0811 22:26:09.707818 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:09.707900 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260811 22:26:09.745016 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 10963
I20260811 22:26:09.785586 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 11393
I20260811 22:26:09.800731 12076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:09.786431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52746 (local address 127.29.106.2:39773)
0811 22:26:09.786594 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:09.786599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:09.786616 (+ 17us) server_negotiation.cc:407] Connection header received
0811 22:26:09.786690 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:09.786694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:09.786753 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:09.786824 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:09.788468 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:09.789207 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:09.791073 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:09.791680 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:09.791786 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:09.794926 (+ 3140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:09.794940 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:09.794954 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:09.794994 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:09.798039 (+ 3045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:09.798801 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:09.798809 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:09.798811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:09.798943 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:09.799358 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:09.799363 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:09.799365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:09.799738 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:09.799994 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:09.800291 (+ 297us) server_negotiation.cc:299] Negotiation successful
0811 22:26:09.800466 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260811 22:26:09.801374 11627 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:26:09.917166 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 11393
I20260811 22:26:09.947037 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 11660
I20260811 22:26:09.959437 12082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:09.947768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58064 (local address 127.29.106.3:38811)
0811 22:26:09.947953 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:09.947958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:09.947975 (+ 17us) server_negotiation.cc:407] Connection header received
0811 22:26:09.948118 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:09.948122 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:09.948204 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:09.948284 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:09.949545 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:09.950273 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:09.951522 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:09.951921 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:09.952047 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:09.954378 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:09.954394 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:09.954412 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:09.954445 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:09.957086 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:09.957649 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:09.957656 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:09.957658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:09.957772 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:09.958049 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:09.958052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:09.958054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:09.958408 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:09.958618 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:09.958854 (+ 236us) server_negotiation.cc:299] Negotiation successful
0811 22:26:09.959086 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260811 22:26:09.960115 11896 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:26:10.056919 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 11660
I20260811 22:26:10.079195 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 10382
I20260811 22:26:10.080325 10705 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:26:10.131794 12047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:10.131346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:46573 (local address 127.0.0.1:39432)
0811 22:26:10.131535 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:10.131630 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260811 22:26:10.132889 12047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:10.132498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:46573 (local address 127.0.0.1:39446)
0811 22:26:10.132663 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:10.132753 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260811 22:26:10.197177 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 10382
I20260811 22:26:10.226452 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 10567
I20260811 22:26:10.227442 11098 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:26:10.349001 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 10567
2026-08-11T22:26:10Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10717 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:26:10 dist-test-slave-gjrz krb5kdc[12150](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:26:10 dist-test-slave-gjrz krb5kdc[12150](info): set up 2 sockets
Aug 11 22:26:10 dist-test-slave-gjrz krb5kdc[12150](info): commencing operation
krb5kdc: starting...
W20260811 22:26:11.639719 30120 mini_kdc.cc:121] Time spent starting KDC: real 1.238s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Aug 11 22:26:11 dist-test-slave-gjrz krb5kdc[12150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:26:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:26:11Z Disabled control of system clock
WARNING: no policy specified for oryx/127.29.106.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.106.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.106.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.62.keytab.
Entry for principal oryx/127.29.106.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.62.keytab.
I20260811 22:26:11.860316 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:35049
--webserver_interface=127.29.106.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.62.keytab
--principal=oryx/127.29.106.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:38537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:35049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:26:12.179390 12170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:26:12.179661 12170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:26:12.179797 12170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:26:12.186781 12170 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:26:12.186872 12170 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260811 22:26:12.186919 12170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:26:12.187013 12170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:26:12.187075 12170 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:26:12.187142 12170 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:26:12.196772 12170 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:38537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:35049
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.62.keytab
--principal=oryx/127.29.106.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:35049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.106.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:26:12.197948 12170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:26:12.199278 12170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:26:12.210350 12184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:12.210389 12179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:12.210443 12177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:26:12.211233 12170 server_base.cc:1034] running on GCE node
I20260811 22:26:12.211915 12170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:26:12.213174 12170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:26:12.214393 12170 hybrid_clock.cc:648] HybridClock initialized: now 1786487172214322 us; error 43 us; skew 500 ppm
I20260811 22:26:12.214711 12170 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:26:12 dist-test-slave-gjrz krb5kdc[12150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487172, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.106.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:26:12.222892 12170 init.cc:377] Logged in from keytab as oryx/127.29.106.62@KRBTEST.COM (short username oryx)
I20260811 22:26:12.224246 12170 webserver.cc:460] Webserver started at http://127.29.106.62:39437/ using document root <none> and password file <none>
I20260811 22:26:12.224862 12170 fs_manager.cc:359] Metadata directory not provided
I20260811 22:26:12.224957 12170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:26:12.225222 12170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:26:12.228091 12170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "0ecd4bd0e93d4f0f84a34aa8ecbe04fe"
format_stamp: "Formatted at 2026-08-11 22:26:12 on dist-test-slave-gjrz"
server_key: "7fdf8bc99be512d2e83ff8521360db19"
server_key_iv: "d9f88b542474e714164af838bafe19ea"
server_key_version: "encryptionkey@0"
I20260811 22:26:12.228842 12170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "0ecd4bd0e93d4f0f84a34aa8ecbe04fe"
format_stamp: "Formatted at 2026-08-11 22:26:12 on dist-test-slave-gjrz"
server_key: "7fdf8bc99be512d2e83ff8521360db19"
server_key_iv: "d9f88b542474e714164af838bafe19ea"
server_key_version: "encryptionkey@0"
I20260811 22:26:12.233642 12170 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260811 22:26:12.237030 12200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:12.238291 12170 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260811 22:26:12.238444 12170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "0ecd4bd0e93d4f0f84a34aa8ecbe04fe"
format_stamp: "Formatted at 2026-08-11 22:26:12 on dist-test-slave-gjrz"
server_key: "7fdf8bc99be512d2e83ff8521360db19"
server_key_iv: "d9f88b542474e714164af838bafe19ea"
server_key_version: "encryptionkey@0"
I20260811 22:26:12.238629 12170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:26:12.258345 12170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:26:12.262247 12170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:26:12.262501 12170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:26:12.282114 12170 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.62:35049
I20260811 22:26:12.282133 12311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.62:35049 every 8 connection(s)
I20260811 22:26:12.283356 12170 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:26:12.286844 12312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:26:12.289346 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 12170
I20260811 22:26:12.289593 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:26:12.290356 30120 external_mini_cluster.cc:1428] Setting key 55f5a1e3b1cf38f8c215d278394af133
I20260811 22:26:12.299062 12312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ecd4bd0e93d4f0f84a34aa8ecbe04fe: Bootstrap starting.
I20260811 22:26:12.302618 12312 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ecd4bd0e93d4f0f84a34aa8ecbe04fe: Neither blocks nor log segments found. Creating new log.
I20260811 22:26:12.303886 12312 log.cc:824] T 00000000000000000000000000000000 P 0ecd4bd0e93d4f0f84a34aa8ecbe04fe: Log is configured to *not* fsync() on all Append() calls
Aug 11 22:26:12 dist-test-slave-gjrz krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM
I20260811 22:26:12.306875 12312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ecd4bd0e93d4f0f84a34aa8ecbe04fe: No bootstrap required, opened a new log
I20260811 22:26:12.313409 12316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:12.292383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41890 (local address 127.29.106.62:35049)
0811 22:26:12.293256 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:12.293268 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:12.293309 (+ 41us) server_negotiation.cc:407] Connection header received
0811 22:26:12.295222 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:12.295248 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:12.295651 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:12.296006 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:12.297656 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:12.298938 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:12.300695 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:12.301427 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:12.301554 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:12.306215 (+ 4661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:12.306226 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:12.306244 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:12.306286 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:12.309394 (+ 3108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:12.310140 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:12.310147 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:12.310152 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:12.310334 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:12.310713 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:12.310717 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:12.310719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:12.311306 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:12.311615 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:12.312050 (+ 435us) server_negotiation.cc:299] Negotiation successful
0811 22:26:12.312436 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":350,"threads_started":1}
I20260811 22:26:12.314465 12312 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ecd4bd0e93d4f0f84a34aa8ecbe04fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ecd4bd0e93d4f0f84a34aa8ecbe04fe" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 35049 } }
I20260811 22:26:12.314795 12312 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ecd4bd0e93d4f0f84a34aa8ecbe04fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:26:12.314890 12312 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ecd4bd0e93d4f0f84a34aa8ecbe04fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ecd4bd0e93d4f0f84a34aa8ecbe04fe, State: Initialized, Role: FOLLOWER
I20260811 22:26:12.315459 12312 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ecd4bd0e93d4f0f84a34aa8ecbe04fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ecd4bd0e93d4f0f84a34aa8ecbe04fe" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 35049 } }
I20260811 22:26:12.315693 12312 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ecd4bd0e93d4f0f84a34aa8ecbe04fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:26:12.315843 12312 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ecd4bd0e93d4f0f84a34aa8ecbe04fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:26:12.316028 12312 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ecd4bd0e93d4f0f84a34aa8ecbe04fe [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:26:12.318038 12312 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ecd4bd0e93d4f0f84a34aa8ecbe04fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ecd4bd0e93d4f0f84a34aa8ecbe04fe" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 35049 } }
I20260811 22:26:12.318509 12312 leader_election.cc:302] T 00000000000000000000000000000000 P 0ecd4bd0e93d4f0f84a34aa8ecbe04fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ecd4bd0e93d4f0f84a34aa8ecbe04fe; no voters:
I20260811 22:26:12.319108 12312 leader_election.cc:288] T 00000000000000000000000000000000 P 0ecd4bd0e93d4f0f84a34aa8ecbe04fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:26:12.319239 12318 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ecd4bd0e93d4f0f84a34aa8ecbe04fe [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:26:12.320011 12318 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ecd4bd0e93d4f0f84a34aa8ecbe04fe [term 1 LEADER]: Becoming Leader. State: Replica: 0ecd4bd0e93d4f0f84a34aa8ecbe04fe, State: Running, Role: LEADER
I20260811 22:26:12.320535 12318 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ecd4bd0e93d4f0f84a34aa8ecbe04fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ecd4bd0e93d4f0f84a34aa8ecbe04fe" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 35049 } }
I20260811 22:26:12.321096 12312 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ecd4bd0e93d4f0f84a34aa8ecbe04fe [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:26:12.323325 12325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ecd4bd0e93d4f0f84a34aa8ecbe04fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ecd4bd0e93d4f0f84a34aa8ecbe04fe. Latest consensus state: current_term: 1 leader_uuid: "0ecd4bd0e93d4f0f84a34aa8ecbe04fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ecd4bd0e93d4f0f84a34aa8ecbe04fe" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 35049 } } }
I20260811 22:26:12.323375 12319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ecd4bd0e93d4f0f84a34aa8ecbe04fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ecd4bd0e93d4f0f84a34aa8ecbe04fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ecd4bd0e93d4f0f84a34aa8ecbe04fe" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 35049 } } }
I20260811 22:26:12.323700 12325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ecd4bd0e93d4f0f84a34aa8ecbe04fe [sys.catalog]: This master's current role is: LEADER
I20260811 22:26:12.323700 12319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ecd4bd0e93d4f0f84a34aa8ecbe04fe [sys.catalog]: This master's current role is: LEADER
I20260811 22:26:12.325171 12332 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:26:12.330534 12332 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:26:12.340551 12332 catalog_manager.cc:1347] Generated new cluster ID: 7cef738901e04ef5b368b69b1a5bd407
I20260811 22:26:12.340663 12332 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:26:12.351311 12332 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:26:12.353209 12332 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:26:12.361860 12332 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ecd4bd0e93d4f0f84a34aa8ecbe04fe: Generated new TSK 0
I20260811 22:26:12.363229 12332 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.29.106.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.106.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.106.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.1.keytab.
Entry for principal oryx/127.29.106.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.1.keytab.
I20260811 22:26:12.416994 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.1:0
--local_ip_for_outbound_sockets=127.29.106.1
--webserver_interface=127.29.106.1
--webserver_port=0
--tserver_master_addrs=127.29.106.62:35049
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.1.keytab
--principal=oryx/127.29.106.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:38537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:26:12.738967 12357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:26:12.739312 12357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:26:12.739476 12357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:26:12.746690 12357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:26:12.746781 12357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:26:12.746904 12357 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 22:26:12.747005 12357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.1
I20260811 22:26:12.756263 12357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:38537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.1.keytab
--principal=oryx/127.29.106.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.106.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.106.62:35049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.1
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:26:12.757350 12357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:26:12.758615 12357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:26:12.771093 12365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:12.771137 12366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:12.771173 12369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:26:12.771975 12357 server_base.cc:1034] running on GCE node
I20260811 22:26:12.772557 12357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:26:12.773352 12357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:26:12.774572 12357 hybrid_clock.cc:648] HybridClock initialized: now 1786487172774509 us; error 37 us; skew 500 ppm
I20260811 22:26:12.774837 12357 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:26:12 dist-test-slave-gjrz krb5kdc[12150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487172, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.106.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:26:12.777758 12357 init.cc:377] Logged in from keytab as oryx/127.29.106.1@KRBTEST.COM (short username oryx)
I20260811 22:26:12.779062 12357 webserver.cc:460] Webserver started at http://127.29.106.1:39353/ using document root <none> and password file <none>
I20260811 22:26:12.779728 12357 fs_manager.cc:359] Metadata directory not provided
I20260811 22:26:12.779816 12357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:26:12.780071 12357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:26:12.782718 12357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-0/data/instance:
uuid: "f3f6fc50d40445e3b7a35e0170a02880"
format_stamp: "Formatted at 2026-08-11 22:26:12 on dist-test-slave-gjrz"
server_key: "5188c34c7eb612ff32c28172e2f42514"
server_key_iv: "d44cd01e6f3e1d9c8a8ce524ea01fd92"
server_key_version: "encryptionkey@0"
I20260811 22:26:12.783463 12357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance:
uuid: "f3f6fc50d40445e3b7a35e0170a02880"
format_stamp: "Formatted at 2026-08-11 22:26:12 on dist-test-slave-gjrz"
server_key: "5188c34c7eb612ff32c28172e2f42514"
server_key_iv: "d44cd01e6f3e1d9c8a8ce524ea01fd92"
server_key_version: "encryptionkey@0"
I20260811 22:26:12.788081 12357 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260811 22:26:12.791167 12383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:12.792394 12357 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260811 22:26:12.792533 12357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-0/wal
uuid: "f3f6fc50d40445e3b7a35e0170a02880"
format_stamp: "Formatted at 2026-08-11 22:26:12 on dist-test-slave-gjrz"
server_key: "5188c34c7eb612ff32c28172e2f42514"
server_key_iv: "d44cd01e6f3e1d9c8a8ce524ea01fd92"
server_key_version: "encryptionkey@0"
I20260811 22:26:12.792718 12357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:26:12.806782 12357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:26:12.810443 12357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:26:12.810668 12357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:26:12.813186 12357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:26:12.813287 12357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:12.813426 12357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:26:12.813498 12357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 22:26:12 dist-test-slave-gjrz krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487172, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.106.1@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM
I20260811 22:26:12.834966 12316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:12.813776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:41859 (local address 127.29.106.62:35049)
0811 22:26:12.813975 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:12.813982 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:12.815316 (+ 1334us) server_negotiation.cc:407] Connection header received
0811 22:26:12.817243 (+ 1927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:12.817249 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:12.817335 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:12.817422 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:12.819450 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:12.820417 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:12.822864 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:12.823462 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:12.823579 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:12.827751 (+ 4172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:12.827759 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:12.827764 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:12.827809 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:12.831099 (+ 3290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:12.832141 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:12.832147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:12.832151 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:12.832304 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:12.832882 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:12.832887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:12.832890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:12.833260 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:12.833412 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:12.834370 (+ 958us) server_negotiation.cc:299] Negotiation successful
0811 22:26:12.834612 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260811 22:26:12.836568 12398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:12.814347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:35049 (local address 127.29.106.1:41859)
0811 22:26:12.815149 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:12.815185 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:12.816980 (+ 1795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:12.817675 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:12.817683 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:12.818456 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:12.819257 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:12.819303 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:12.820574 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:12.820580 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:12.822729 (+ 2149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:12.822742 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:12.823710 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:12.823719 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:12.823920 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:12.824685 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:12.824706 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:12.827499 (+ 2793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:26:12.831314 (+ 3815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:12.831327 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:12.831341 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:12.831988 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:12.832446 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:12.832451 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:12.832455 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:12.832749 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:12.833435 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:26:12.833443 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:26:12.833898 (+ 455us) client_negotiation.cc:769] Sending connection context
0811 22:26:12.834250 (+ 352us) client_negotiation.cc:240] Negotiation successful
0811 22:26:12.834887 (+ 637us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":338,"threads_started":1}
I20260811 22:26:12.852419 12357 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.1:33759
I20260811 22:26:12.852439 12602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.1:33759 every 8 connection(s)
I20260811 22:26:12.853878 12357 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
I20260811 22:26:12.855916 12604 heartbeater.cc:347] Connected to a master server at 127.29.106.62:35049
I20260811 22:26:12.856134 12604 heartbeater.cc:464] Registering TS with master...
I20260811 22:26:12.856750 12604 heartbeater.cc:511] Master 127.29.106.62:35049 requested a full tablet report, sending...
I20260811 22:26:12.857771 12316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:12.843614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:35279 (local address 127.29.106.62:35049)
0811 22:26:12.843774 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:12.843779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:12.844659 (+ 880us) server_negotiation.cc:407] Connection header received
0811 22:26:12.844779 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:12.844784 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:12.844860 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:12.844957 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:12.846342 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:12.847465 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:12.849053 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:12.849679 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:12.849965 (+ 286us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:12.851858 (+ 1893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:12.851864 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:12.851868 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:12.851906 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:12.855104 (+ 3198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:12.855855 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:12.855858 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:12.855860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:12.855984 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:12.856382 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:12.856387 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:12.856391 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:12.856640 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:12.856767 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:12.857215 (+ 448us) server_negotiation.cc:299] Negotiation successful
0811 22:26:12.857415 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260811 22:26:12.857816 12564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:12.843710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:35049 (local address 127.29.106.1:35279)
0811 22:26:12.844560 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:12.844578 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:12.844687 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:12.845008 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:12.845013 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:12.845628 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:12.846181 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:12.846199 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:12.847667 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:12.847676 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:12.848922 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:12.848935 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:12.849972 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:12.849987 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:12.850164 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:12.850759 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:12.850775 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:12.851673 (+ 898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:26:12.855228 (+ 3555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:12.855233 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:12.855236 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:12.855659 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:12.856119 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:12.856128 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:12.856132 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:12.856281 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:12.856756 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:26:12.856761 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:26:12.856970 (+ 209us) client_negotiation.cc:769] Sending connection context
0811 22:26:12.857089 (+ 119us) client_negotiation.cc:240] Negotiation successful
0811 22:26:12.857406 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"thread_start_us":586,"threads_started":1}
I20260811 22:26:12.858865 12228 ts_manager.cc:194] Registered new tserver with Master: f3f6fc50d40445e3b7a35e0170a02880 (127.29.106.1:33759)
I20260811 22:26:12.859155 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 12357
I20260811 22:26:12.859395 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance
I20260811 22:26:12.860185 30120 external_mini_cluster.cc:1428] Setting key 7ba2e966549c38d518e8ab58c8de0f3e
I20260811 22:26:12.860523 12228 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.106.1@KRBTEST.COM'} at 127.29.106.1:41859
WARNING: no policy specified for oryx/127.29.106.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.106.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.106.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.2.keytab.
Entry for principal oryx/127.29.106.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.2.keytab.
I20260811 22:26:12.912645 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.2:0
--local_ip_for_outbound_sockets=127.29.106.2
--webserver_interface=127.29.106.2
--webserver_port=0
--tserver_master_addrs=127.29.106.62:35049
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.2.keytab
--principal=oryx/127.29.106.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:38537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:26:13.231595 12609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:26:13.231838 12609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:26:13.231969 12609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:26:13.239027 12609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:26:13.239121 12609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:26:13.239224 12609 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 22:26:13.239336 12609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.2
I20260811 22:26:13.248731 12609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:38537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.2.keytab
--principal=oryx/127.29.106.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.106.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.106.62:35049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.2
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:26:13.249874 12609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:26:13.251104 12609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:26:13.263402 12623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:13.263434 12625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:13.263988 12630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:26:13.264199 12609 server_base.cc:1034] running on GCE node
I20260811 22:26:13.264760 12609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:26:13.265555 12609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:26:13.266762 12609 hybrid_clock.cc:648] HybridClock initialized: now 1786487173266699 us; error 39 us; skew 500 ppm
I20260811 22:26:13.267004 12609 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:26:13 dist-test-slave-gjrz krb5kdc[12150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487173, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.106.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:26:13.269958 12609 init.cc:377] Logged in from keytab as oryx/127.29.106.2@KRBTEST.COM (short username oryx)
I20260811 22:26:13.271234 12609 webserver.cc:460] Webserver started at http://127.29.106.2:33385/ using document root <none> and password file <none>
I20260811 22:26:13.271883 12609 fs_manager.cc:359] Metadata directory not provided
I20260811 22:26:13.271975 12609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:26:13.272279 12609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:26:13.274966 12609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-1/data/instance:
uuid: "d57268eda08f4b679ade15c2c4b251c9"
format_stamp: "Formatted at 2026-08-11 22:26:13 on dist-test-slave-gjrz"
server_key: "e036df1e27cae80b0dace28410acba49"
server_key_iv: "efdb826a131e8b89ad8c175a03711920"
server_key_version: "encryptionkey@0"
I20260811 22:26:13.275817 12609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance:
uuid: "d57268eda08f4b679ade15c2c4b251c9"
format_stamp: "Formatted at 2026-08-11 22:26:13 on dist-test-slave-gjrz"
server_key: "e036df1e27cae80b0dace28410acba49"
server_key_iv: "efdb826a131e8b89ad8c175a03711920"
server_key_version: "encryptionkey@0"
I20260811 22:26:13.280514 12609 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 22:26:13.283888 12640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:13.285152 12609 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 22:26:13.285300 12609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-1/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-1/wal
uuid: "d57268eda08f4b679ade15c2c4b251c9"
format_stamp: "Formatted at 2026-08-11 22:26:13 on dist-test-slave-gjrz"
server_key: "e036df1e27cae80b0dace28410acba49"
server_key_iv: "efdb826a131e8b89ad8c175a03711920"
server_key_version: "encryptionkey@0"
I20260811 22:26:13.285491 12609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:26:13.333462 12609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:26:13.337343 12609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:26:13.337589 12609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:26:13.340348 12609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:26:13.340446 12609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:13.340642 12609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:26:13.340848 12609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Aug 11 22:26:13 dist-test-slave-gjrz krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487173, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.106.2@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM
I20260811 22:26:13.363193 12659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:13.341572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:35049 (local address 127.29.106.2:54543)
0811 22:26:13.342402 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:13.342442 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:13.344319 (+ 1877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:13.345059 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:13.345070 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:13.345831 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:13.346636 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:13.346659 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:13.348235 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:13.348239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:13.350377 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:13.350409 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:13.351257 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:13.351285 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:13.351584 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:13.352289 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:13.352310 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:13.355381 (+ 3071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:26:13.358691 (+ 3310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:13.358700 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:13.358721 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:13.359322 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:13.359755 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:13.359760 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:13.359763 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:13.360055 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:13.360686 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:26:13.360693 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:26:13.361063 (+ 370us) client_negotiation.cc:769] Sending connection context
0811 22:26:13.361406 (+ 343us) client_negotiation.cc:240] Negotiation successful
0811 22:26:13.361916 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":343,"threads_started":1}
I20260811 22:26:13.364101 12316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:13.340848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:54543 (local address 127.29.106.62:35049)
0811 22:26:13.341049 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:13.341053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:13.342550 (+ 1497us) server_negotiation.cc:407] Connection header received
0811 22:26:13.344624 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:13.344631 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:13.344714 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:13.344830 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:13.346875 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:13.348060 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:13.350582 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:13.351005 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:13.351124 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:13.355653 (+ 4529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:13.355660 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:13.355665 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:13.355708 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:13.358510 (+ 2802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:13.359496 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:13.359500 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:13.359502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:13.359624 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:13.360180 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:13.360185 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:13.360189 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:13.360551 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:13.360696 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:13.363682 (+ 2986us) server_negotiation.cc:299] Negotiation successful
0811 22:26:13.363873 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260811 22:26:13.382997 12791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:13.369714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:35049 (local address 127.29.106.2:50885)
0811 22:26:13.370357 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:13.370376 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:13.370479 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:13.370769 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:13.370772 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:13.371195 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:13.371609 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:13.371624 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:13.372780 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:13.372785 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:13.374319 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:13.374335 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:13.375254 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:13.375259 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:13.375489 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:13.376074 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:13.376096 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:13.376969 (+ 873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:26:13.380287 (+ 3318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:13.380292 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:13.380296 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:13.380868 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:13.381298 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:13.381303 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:13.381307 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:13.381433 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:13.381975 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:26:13.381980 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:26:13.382234 (+ 254us) client_negotiation.cc:769] Sending connection context
0811 22:26:13.382417 (+ 183us) client_negotiation.cc:240] Negotiation successful
0811 22:26:13.382681 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":361,"threads_started":1}
I20260811 22:26:13.383170 12316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:13.369565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:50885 (local address 127.29.106.62:35049)
0811 22:26:13.369746 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:13.369752 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:13.370457 (+ 705us) server_negotiation.cc:407] Connection header received
0811 22:26:13.370574 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:13.370579 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:13.370658 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:13.370762 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:13.371749 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:13.372653 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:13.374457 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:13.374928 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:13.375047 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:13.377143 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:13.377150 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:13.377154 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:13.377193 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:13.380131 (+ 2938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:13.381064 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:13.381069 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:13.381073 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:13.381187 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:13.381532 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:13.381537 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:13.381541 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:13.381831 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:13.381996 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:13.382553 (+ 557us) server_negotiation.cc:299] Negotiation successful
0811 22:26:13.382798 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260811 22:26:13.387907 12609 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.2:39563
I20260811 22:26:13.387948 12858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.2:39563 every 8 connection(s)
I20260811 22:26:13.389020 12609 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
I20260811 22:26:13.390808 12859 heartbeater.cc:347] Connected to a master server at 127.29.106.62:35049
I20260811 22:26:13.391011 12859 heartbeater.cc:464] Registering TS with master...
I20260811 22:26:13.391628 12859 heartbeater.cc:511] Master 127.29.106.62:35049 requested a full tablet report, sending...
I20260811 22:26:13.392796 12227 ts_manager.cc:194] Registered new tserver with Master: d57268eda08f4b679ade15c2c4b251c9 (127.29.106.2:39563)
I20260811 22:26:13.393597 12227 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.106.2@KRBTEST.COM'} at 127.29.106.2:54543
I20260811 22:26:13.394534 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 12609
I20260811 22:26:13.394701 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance
I20260811 22:26:13.395131 30120 external_mini_cluster.cc:1428] Setting key ca1cf5340de0c2212786c8ae3a869063
WARNING: no policy specified for oryx/127.29.106.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.106.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.106.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.3.keytab.
Entry for principal oryx/127.29.106.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.3.keytab.
I20260811 22:26:13.446452 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.3:0
--local_ip_for_outbound_sockets=127.29.106.3
--webserver_interface=127.29.106.3
--webserver_port=0
--tserver_master_addrs=127.29.106.62:35049
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.3.keytab
--principal=oryx/127.29.106.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:38537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:26:13.771157 12863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:26:13.771457 12863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:26:13.771605 12863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:26:13.778538 12863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:26:13.778641 12863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:26:13.778757 12863 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 22:26:13.778864 12863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.3
I20260811 22:26:13.788156 12863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:38537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.3.keytab
--principal=oryx/127.29.106.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.106.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.106.62:35049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.3
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:26:13.789325 12863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:26:13.790609 12863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:26:13.802872 12872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:13.802912 12873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:13.802935 12875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:26:13.803943 12863 server_base.cc:1034] running on GCE node
I20260811 22:26:13.804536 12863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:26:13.805361 12863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:26:13.806566 12863 hybrid_clock.cc:648] HybridClock initialized: now 1786487173806514 us; error 41 us; skew 500 ppm
I20260811 22:26:13.806829 12863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:26:13 dist-test-slave-gjrz krb5kdc[12150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487173, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.106.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:26:13.809695 12863 init.cc:377] Logged in from keytab as oryx/127.29.106.3@KRBTEST.COM (short username oryx)
I20260811 22:26:13.810958 12863 webserver.cc:460] Webserver started at http://127.29.106.3:44373/ using document root <none> and password file <none>
I20260811 22:26:13.811647 12863 fs_manager.cc:359] Metadata directory not provided
I20260811 22:26:13.811749 12863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:26:13.812048 12863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:26:13.814736 12863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-2/data/instance:
uuid: "8f65ed2870d3440097c9b56016f85408"
format_stamp: "Formatted at 2026-08-11 22:26:13 on dist-test-slave-gjrz"
server_key: "c9c8314dd7d01fadfd236f3a301d0c86"
server_key_iv: "03a2dd9ed13591738645f002f199533d"
server_key_version: "encryptionkey@0"
I20260811 22:26:13.815529 12863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance:
uuid: "8f65ed2870d3440097c9b56016f85408"
format_stamp: "Formatted at 2026-08-11 22:26:13 on dist-test-slave-gjrz"
server_key: "c9c8314dd7d01fadfd236f3a301d0c86"
server_key_iv: "03a2dd9ed13591738645f002f199533d"
server_key_version: "encryptionkey@0"
I20260811 22:26:13.820235 12863 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260811 22:26:13.823539 12887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:13.824733 12863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260811 22:26:13.824878 12863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-2/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-2/wal
uuid: "8f65ed2870d3440097c9b56016f85408"
format_stamp: "Formatted at 2026-08-11 22:26:13 on dist-test-slave-gjrz"
server_key: "c9c8314dd7d01fadfd236f3a301d0c86"
server_key_iv: "03a2dd9ed13591738645f002f199533d"
server_key_version: "encryptionkey@0"
I20260811 22:26:13.825075 12863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:26:13.854333 12863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:26:13.858865 12863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:26:13.859105 12863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:26:13.861541 12863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:26:13.861639 12863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:13.861778 12863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:26:13.861846 12863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:13.863663 12604 heartbeater.cc:503] Master 127.29.106.62:35049 was elected leader, sending a full tablet report...
Aug 11 22:26:13 dist-test-slave-gjrz krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487173, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.106.3@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM
I20260811 22:26:13.885062 12316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:13.862254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:44047 (local address 127.29.106.62:35049)
0811 22:26:13.862457 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:13.862462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:13.864292 (+ 1830us) server_negotiation.cc:407] Connection header received
0811 22:26:13.866653 (+ 2361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:13.866658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:13.866736 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:13.866845 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:13.868948 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:13.869961 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:13.872855 (+ 2894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:13.873415 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:13.873539 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:13.878080 (+ 4541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:13.878085 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:13.878088 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:13.878119 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:13.881469 (+ 3350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:13.882548 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:13.882554 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:13.882557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:13.882714 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:13.883317 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:13.883321 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:13.883324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:13.883656 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:13.883809 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:13.884485 (+ 676us) server_negotiation.cc:299] Negotiation successful
0811 22:26:13.884725 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260811 22:26:13.885943 12900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:13.863105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:35049 (local address 127.29.106.3:44047)
0811 22:26:13.864109 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:13.864172 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:13.866362 (+ 2190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:13.867113 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:13.867124 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:13.867852 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:13.868755 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:13.868787 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:13.870129 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:13.870136 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:13.872694 (+ 2558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:13.872712 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:13.873813 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:13.873821 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:13.874110 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:13.874854 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:13.874886 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:13.877829 (+ 2943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:26:13.881662 (+ 3833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:13.881674 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:13.881693 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:13.882398 (+ 705us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:13.882833 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:13.882838 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:13.882841 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:13.883154 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:13.883791 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:26:13.883798 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:26:13.884134 (+ 336us) client_negotiation.cc:769] Sending connection context
0811 22:26:13.884358 (+ 224us) client_negotiation.cc:240] Negotiation successful
0811 22:26:13.884831 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":418,"threads_started":1}
I20260811 22:26:13.909694 13014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:13.891727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:35049 (local address 127.29.106.3:55613)
0811 22:26:13.895857 (+ 4130us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:13.895878 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:13.896016 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:13.896415 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:13.896420 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:13.897005 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:13.897685 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:13.897703 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:13.899013 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:13.899018 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:13.900834 (+ 1816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:13.900855 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:13.901725 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:13.901733 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:13.901905 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:13.902425 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:13.902443 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:13.903159 (+ 716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:26:13.906614 (+ 3455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:13.906619 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:13.906623 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:13.907173 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:13.907719 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:13.907725 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:13.907729 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:13.907911 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:13.908487 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:26:13.908496 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:26:13.908688 (+ 192us) client_negotiation.cc:769] Sending connection context
0811 22:26:13.908841 (+ 153us) client_negotiation.cc:240] Negotiation successful
0811 22:26:13.909189 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3966,"thread_start_us":3808,"threads_started":1}
I20260811 22:26:13.910475 12316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:13.892122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:55613 (local address 127.29.106.62:35049)
0811 22:26:13.892288 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:13.892294 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:13.895952 (+ 3658us) server_negotiation.cc:407] Connection header received
0811 22:26:13.896182 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:13.896188 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:13.896271 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:13.896356 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:13.897852 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:13.898902 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:13.900973 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:13.901565 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:13.901681 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:13.903741 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:13.903747 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:13.903752 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:13.903788 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:13.906490 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:13.907352 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:13.907357 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:13.907360 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:13.907564 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:13.908040 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:13.908045 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:13.908049 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:13.908368 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:13.908517 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:13.910009 (+ 1492us) server_negotiation.cc:299] Negotiation successful
0811 22:26:13.910223 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260811 22:26:13.914734 12863 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.3:38153
I20260811 22:26:13.914754 13103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.3:38153 every 8 connection(s)
I20260811 22:26:13.915882 12863 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
I20260811 22:26:13.917649 13106 heartbeater.cc:347] Connected to a master server at 127.29.106.62:35049
I20260811 22:26:13.917862 13106 heartbeater.cc:464] Registering TS with master...
I20260811 22:26:13.917991 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 12863
I20260811 22:26:13.918188 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance
I20260811 22:26:13.918480 13106 heartbeater.cc:511] Master 127.29.106.62:35049 requested a full tablet report, sending...
I20260811 22:26:13.918609 30120 external_mini_cluster.cc:1428] Setting key e3e21b67fdfa3587d70945101a3726ac
I20260811 22:26:13.919891 12228 ts_manager.cc:194] Registered new tserver with Master: 8f65ed2870d3440097c9b56016f85408 (127.29.106.3:38153)
I20260811 22:26:13.920845 12228 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.106.3@KRBTEST.COM'} at 127.29.106.3:44047
I20260811 22:26:13.921419 30120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 11 22:26:13 dist-test-slave-gjrz krb5kdc[12150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487173, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 11 22:26:13 dist-test-slave-gjrz krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:13 dist-test-slave-gjrz krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:26:14.395874 12859 heartbeater.cc:503] Master 127.29.106.62:35049 was elected leader, sending a full tablet report...
I20260811 22:26:14.923421 13106 heartbeater.cc:503] Master 127.29.106.62:35049 was elected leader, sending a full tablet report...
W20260811 22:26:14.955646 13118 sasl_common.cc:323] Time spent calling sasl_client_start(): real 0.985s user 0.007s sys 0.010s
W20260811 22:26:14.956797 12316 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:13.965589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41904 (local address 127.29.106.62:35049)
0811 22:26:13.965779 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:13.965785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:13.965976 (+ 191us) server_negotiation.cc:407] Connection header received
0811 22:26:13.966172 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:13.966176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:13.966240 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:13.966321 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:13.967723 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:13.968458 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:13.969645 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:13.970046 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:13.970132 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:14.956567 (+986435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41904: BlockingRecv error: recv got EOF from 127.0.0.1:41904 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 11 22:26:14 dist-test-slave-gjrz krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487173, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM
I20260811 22:26:14.973824 12316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:14.961089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41914 (local address 127.29.106.62:35049)
0811 22:26:14.961237 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:14.961241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:14.961569 (+ 328us) server_negotiation.cc:407] Connection header received
0811 22:26:14.961720 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:14.961723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:14.961798 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:14.961895 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:14.963033 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:14.963812 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:14.965109 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:14.965504 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:14.965604 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:14.969346 (+ 3742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:14.969351 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:14.969354 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:14.969386 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:14.971763 (+ 2377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:14.972330 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:14.972333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:14.972336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:14.972445 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:14.972721 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:14.972724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:14.972726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:14.972973 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:14.973146 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:14.973360 (+ 214us) server_negotiation.cc:299] Negotiation successful
0811 22:26:14.973521 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260811 22:26:14.977626 12228 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41914:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:26:14.984383 12228 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:26:15.006549 13139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:14.995864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41724 (local address 127.29.106.1:33759)
0811 22:26:14.997678 (+ 1814us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:14.997684 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:14.997707 (+ 23us) server_negotiation.cc:407] Connection header received
0811 22:26:14.997814 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:14.997822 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:14.998060 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:14.998995 (+ 935us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:26:14.999228 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:15.001293 (+ 2065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:15.004010 (+ 2717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:15.005404 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:15.005551 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:15.005751 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:15.006089 (+ 338us) server_negotiation.cc:299] Negotiation successful
0811 22:26:15.006257 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1672,"thread_start_us":1541,"threads_started":1}
I20260811 22:26:15.006626 13141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:14.995702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:33759 (local address 127.0.0.1:41724)
0811 22:26:14.997006 (+ 1304us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:14.997066 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:14.997230 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:14.998314 (+ 1084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:14.998322 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:14.998356 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:26:14.998788 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:14.998804 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:15.001443 (+ 2639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:15.001449 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:15.003861 (+ 2412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:15.003873 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:15.005628 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:15.005634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:15.005835 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:15.005885 (+ 50us) client_negotiation.cc:769] Sending connection context
0811 22:26:15.006007 (+ 122us) client_negotiation.cc:240] Negotiation successful
0811 22:26:15.006195 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1020,"thread_start_us":1658,"threads_started":1}
I20260811 22:26:15.009618 13145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:14.997671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.3:38153 (local address 127.0.0.1:50906)
0811 22:26:14.998150 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:14.998167 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:14.998276 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:15.002530 (+ 4254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:15.002539 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:15.002563 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:26:15.003032 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:15.003045 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:15.004820 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:15.004825 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:15.007310 (+ 2485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:15.007329 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:15.008848 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:15.008855 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:15.009007 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:15.009038 (+ 31us) client_negotiation.cc:769] Sending connection context
0811 22:26:15.009141 (+ 103us) client_negotiation.cc:240] Negotiation successful
0811 22:26:15.009262 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":1,"thread_start_us":235,"threads_started":1}
I20260811 22:26:15.009650 13146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:14.997928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50906 (local address 127.29.106.3:38153)
0811 22:26:15.001829 (+ 3901us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:15.001835 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:15.001858 (+ 23us) server_negotiation.cc:407] Connection header received
0811 22:26:15.001945 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:15.001950 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:15.002116 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:15.002300 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:26:15.003218 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:15.004624 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:15.007468 (+ 2844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:15.008630 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:15.008773 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:15.008963 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:15.009192 (+ 229us) server_negotiation.cc:299] Negotiation successful
0811 22:26:15.009355 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3764,"thread_start_us":3638,"threads_started":1}
I20260811 22:26:15.010567 12470 tablet_service.cc:1467] Processing CreateTablet for tablet 29b4025a5155470a9d4be2444635f713 (DEFAULT_TABLE table=test-table [id=cb4939fc17c24d7fb46a4c837bda6d78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:26:15.011216 13137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:14.996209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.2:39563 (local address 127.0.0.1:41924)
0811 22:26:14.997110 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:14.997131 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:14.997257 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:15.004260 (+ 7003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:15.004265 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:15.004290 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:26:15.004612 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:15.004625 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:15.006270 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:15.006274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:15.008357 (+ 2083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:15.008371 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:15.010491 (+ 2120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:15.010497 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:15.010674 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:15.010707 (+ 33us) client_negotiation.cc:769] Sending connection context
0811 22:26:15.010789 (+ 82us) client_negotiation.cc:240] Negotiation successful
0811 22:26:15.010911 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":560,"threads_started":1}
I20260811 22:26:15.011946 13144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:14.996596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41924 (local address 127.29.106.2:39563)
0811 22:26:15.003727 (+ 7131us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:15.003735 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:15.003763 (+ 28us) server_negotiation.cc:407] Connection header received
0811 22:26:15.003850 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:15.003857 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:15.004076 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:15.004260 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:26:15.004809 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:15.006109 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:15.008573 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:15.009968 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:15.010157 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:15.010332 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:15.010851 (+ 519us) server_negotiation.cc:299] Negotiation successful
0811 22:26:15.011027 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6962,"thread_start_us":6822,"threads_started":1}
I20260811 22:26:15.012102 12470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29b4025a5155470a9d4be2444635f713. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:26:15.013331 12968 tablet_service.cc:1467] Processing CreateTablet for tablet 29b4025a5155470a9d4be2444635f713 (DEFAULT_TABLE table=test-table [id=cb4939fc17c24d7fb46a4c837bda6d78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:26:15.014632 12968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29b4025a5155470a9d4be2444635f713. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:26:15.014642 12732 tablet_service.cc:1467] Processing CreateTablet for tablet 29b4025a5155470a9d4be2444635f713 (DEFAULT_TABLE table=test-table [id=cb4939fc17c24d7fb46a4c837bda6d78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:26:15.015738 12732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29b4025a5155470a9d4be2444635f713. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:26:15.026551 13149 tablet_bootstrap.cc:492] T 29b4025a5155470a9d4be2444635f713 P f3f6fc50d40445e3b7a35e0170a02880: Bootstrap starting.
I20260811 22:26:15.028501 13151 tablet_bootstrap.cc:492] T 29b4025a5155470a9d4be2444635f713 P d57268eda08f4b679ade15c2c4b251c9: Bootstrap starting.
I20260811 22:26:15.030025 13149 tablet_bootstrap.cc:654] T 29b4025a5155470a9d4be2444635f713 P f3f6fc50d40445e3b7a35e0170a02880: Neither blocks nor log segments found. Creating new log.
I20260811 22:26:15.030983 13149 log.cc:824] T 29b4025a5155470a9d4be2444635f713 P f3f6fc50d40445e3b7a35e0170a02880: Log is configured to *not* fsync() on all Append() calls
I20260811 22:26:15.031960 13152 tablet_bootstrap.cc:492] T 29b4025a5155470a9d4be2444635f713 P 8f65ed2870d3440097c9b56016f85408: Bootstrap starting.
I20260811 22:26:15.032122 13151 tablet_bootstrap.cc:654] T 29b4025a5155470a9d4be2444635f713 P d57268eda08f4b679ade15c2c4b251c9: Neither blocks nor log segments found. Creating new log.
I20260811 22:26:15.033378 13151 log.cc:824] T 29b4025a5155470a9d4be2444635f713 P d57268eda08f4b679ade15c2c4b251c9: Log is configured to *not* fsync() on all Append() calls
I20260811 22:26:15.035240 13152 tablet_bootstrap.cc:654] T 29b4025a5155470a9d4be2444635f713 P 8f65ed2870d3440097c9b56016f85408: Neither blocks nor log segments found. Creating new log.
I20260811 22:26:15.035737 13149 tablet_bootstrap.cc:492] T 29b4025a5155470a9d4be2444635f713 P f3f6fc50d40445e3b7a35e0170a02880: No bootstrap required, opened a new log
I20260811 22:26:15.036010 13149 ts_tablet_manager.cc:1397] T 29b4025a5155470a9d4be2444635f713 P f3f6fc50d40445e3b7a35e0170a02880: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260811 22:26:15.036500 13152 log.cc:824] T 29b4025a5155470a9d4be2444635f713 P 8f65ed2870d3440097c9b56016f85408: Log is configured to *not* fsync() on all Append() calls
I20260811 22:26:15.037397 13151 tablet_bootstrap.cc:492] T 29b4025a5155470a9d4be2444635f713 P d57268eda08f4b679ade15c2c4b251c9: No bootstrap required, opened a new log
I20260811 22:26:15.037676 13151 ts_tablet_manager.cc:1397] T 29b4025a5155470a9d4be2444635f713 P d57268eda08f4b679ade15c2c4b251c9: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260811 22:26:15.039389 13152 tablet_bootstrap.cc:492] T 29b4025a5155470a9d4be2444635f713 P 8f65ed2870d3440097c9b56016f85408: No bootstrap required, opened a new log
I20260811 22:26:15.039630 13152 ts_tablet_manager.cc:1397] T 29b4025a5155470a9d4be2444635f713 P 8f65ed2870d3440097c9b56016f85408: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260811 22:26:15.045008 13151 raft_consensus.cc:348] T 29b4025a5155470a9d4be2444635f713 P d57268eda08f4b679ade15c2c4b251c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } }
I20260811 22:26:15.045320 13151 raft_consensus.cc:374] T 29b4025a5155470a9d4be2444635f713 P d57268eda08f4b679ade15c2c4b251c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:26:15.045414 13151 raft_consensus.cc:729] T 29b4025a5155470a9d4be2444635f713 P d57268eda08f4b679ade15c2c4b251c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d57268eda08f4b679ade15c2c4b251c9, State: Initialized, Role: FOLLOWER
I20260811 22:26:15.045955 13151 consensus_queue.cc:260] T 29b4025a5155470a9d4be2444635f713 P d57268eda08f4b679ade15c2c4b251c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } }
I20260811 22:26:15.045991 13149 raft_consensus.cc:348] T 29b4025a5155470a9d4be2444635f713 P f3f6fc50d40445e3b7a35e0170a02880 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } }
I20260811 22:26:15.046407 13149 raft_consensus.cc:374] T 29b4025a5155470a9d4be2444635f713 P f3f6fc50d40445e3b7a35e0170a02880 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:26:15.046523 13149 raft_consensus.cc:729] T 29b4025a5155470a9d4be2444635f713 P f3f6fc50d40445e3b7a35e0170a02880 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3f6fc50d40445e3b7a35e0170a02880, State: Initialized, Role: FOLLOWER
I20260811 22:26:15.047640 13151 ts_tablet_manager.cc:1428] T 29b4025a5155470a9d4be2444635f713 P d57268eda08f4b679ade15c2c4b251c9: Time spent starting tablet: real 0.010s user 0.005s sys 0.006s
I20260811 22:26:15.048353 13149 consensus_queue.cc:260] T 29b4025a5155470a9d4be2444635f713 P f3f6fc50d40445e3b7a35e0170a02880 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } }
I20260811 22:26:15.049403 13152 raft_consensus.cc:348] T 29b4025a5155470a9d4be2444635f713 P 8f65ed2870d3440097c9b56016f85408 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } }
I20260811 22:26:15.049791 13152 raft_consensus.cc:374] T 29b4025a5155470a9d4be2444635f713 P 8f65ed2870d3440097c9b56016f85408 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:26:15.049881 13152 raft_consensus.cc:729] T 29b4025a5155470a9d4be2444635f713 P 8f65ed2870d3440097c9b56016f85408 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f65ed2870d3440097c9b56016f85408, State: Initialized, Role: FOLLOWER
I20260811 22:26:15.050027 13149 ts_tablet_manager.cc:1428] T 29b4025a5155470a9d4be2444635f713 P f3f6fc50d40445e3b7a35e0170a02880: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260811 22:26:15.050539 13152 consensus_queue.cc:260] T 29b4025a5155470a9d4be2444635f713 P 8f65ed2870d3440097c9b56016f85408 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } }
I20260811 22:26:15.052481 13152 ts_tablet_manager.cc:1428] T 29b4025a5155470a9d4be2444635f713 P 8f65ed2870d3440097c9b56016f85408: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260811 22:26:15.100670 13160 raft_consensus.cc:482] T 29b4025a5155470a9d4be2444635f713 P 8f65ed2870d3440097c9b56016f85408 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:26:15.100950 13160 raft_consensus.cc:504] T 29b4025a5155470a9d4be2444635f713 P 8f65ed2870d3440097c9b56016f85408 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } }
I20260811 22:26:15.102535 13160 leader_election.cc:288] T 29b4025a5155470a9d4be2444635f713 P 8f65ed2870d3440097c9b56016f85408 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3f6fc50d40445e3b7a35e0170a02880 (127.29.106.1:33759), d57268eda08f4b679ade15c2c4b251c9 (127.29.106.2:39563)
W20260811 22:26:15.106784 12605 tablet.cc:2369] T 29b4025a5155470a9d4be2444635f713 P f3f6fc50d40445e3b7a35e0170a02880: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:26:15.110569 13144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:15.102983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:35047 (local address 127.29.106.2:39563)
0811 22:26:15.103171 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:15.103177 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:15.103894 (+ 717us) server_negotiation.cc:407] Connection header received
0811 22:26:15.104040 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:15.104044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:15.104120 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:15.104193 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:26:15.104769 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:15.105813 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:15.108293 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:15.109506 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:15.109677 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:15.109751 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:15.110195 (+ 444us) server_negotiation.cc:299] Negotiation successful
0811 22:26:15.110318 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260811 22:26:15.110565 13139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:15.102918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:32847 (local address 127.29.106.1:33759)
0811 22:26:15.103116 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:15.103122 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:15.103847 (+ 725us) server_negotiation.cc:407] Connection header received
0811 22:26:15.104340 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:15.104343 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:15.104408 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:15.104471 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:26:15.105072 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:15.106095 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:15.108406 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:15.109523 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:15.109677 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:15.109751 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:15.110192 (+ 441us) server_negotiation.cc:299] Negotiation successful
0811 22:26:15.110318 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260811 22:26:15.110824 13162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:15.102805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.2:39563 (local address 127.29.106.3:35047)
0811 22:26:15.103734 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:15.103749 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:15.103918 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:15.104225 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:15.104228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:15.104257 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:26:15.104605 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:15.104617 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:15.105937 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:15.105940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:15.108129 (+ 2189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:15.108146 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:15.109747 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:15.109758 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:15.110024 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:15.110064 (+ 40us) client_negotiation.cc:769] Sending connection context
0811 22:26:15.110154 (+ 90us) client_negotiation.cc:240] Negotiation successful
0811 22:26:15.110359 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"mutex_wait_us":89,"spinlock_wait_cycles":13056,"thread_start_us":627,"threads_started":1}
I20260811 22:26:15.110868 13161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:15.102895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:33759 (local address 127.29.106.3:32847)
0811 22:26:15.103729 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:15.103754 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:15.103918 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:15.104523 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:15.104527 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:15.104554 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:26:15.104891 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:15.104900 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:15.106219 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:15.106224 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:15.108251 (+ 2027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:15.108263 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:15.109770 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:15.109775 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:15.110019 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:15.110069 (+ 50us) client_negotiation.cc:769] Sending connection context
0811 22:26:15.110156 (+ 87us) client_negotiation.cc:240] Negotiation successful
0811 22:26:15.110360 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"mutex_wait_us":73,"spinlock_wait_cycles":30336,"thread_start_us":434,"threads_started":1}
I20260811 22:26:15.111837 12509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b4025a5155470a9d4be2444635f713" candidate_uuid: "8f65ed2870d3440097c9b56016f85408" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3f6fc50d40445e3b7a35e0170a02880" is_pre_election: true
I20260811 22:26:15.111923 12765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b4025a5155470a9d4be2444635f713" candidate_uuid: "8f65ed2870d3440097c9b56016f85408" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d57268eda08f4b679ade15c2c4b251c9" is_pre_election: true
I20260811 22:26:15.112294 12509 raft_consensus.cc:2454] T 29b4025a5155470a9d4be2444635f713 P f3f6fc50d40445e3b7a35e0170a02880 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f65ed2870d3440097c9b56016f85408 in term 0.
I20260811 22:26:15.112363 12765 raft_consensus.cc:2454] T 29b4025a5155470a9d4be2444635f713 P d57268eda08f4b679ade15c2c4b251c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f65ed2870d3440097c9b56016f85408 in term 0.
I20260811 22:26:15.112910 12892 leader_election.cc:302] T 29b4025a5155470a9d4be2444635f713 P 8f65ed2870d3440097c9b56016f85408 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f65ed2870d3440097c9b56016f85408, f3f6fc50d40445e3b7a35e0170a02880; no voters:
I20260811 22:26:15.113268 13160 raft_consensus.cc:2781] T 29b4025a5155470a9d4be2444635f713 P 8f65ed2870d3440097c9b56016f85408 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:26:15.113392 13160 raft_consensus.cc:482] T 29b4025a5155470a9d4be2444635f713 P 8f65ed2870d3440097c9b56016f85408 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:26:15.113487 13160 raft_consensus.cc:3037] T 29b4025a5155470a9d4be2444635f713 P 8f65ed2870d3440097c9b56016f85408 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:26:15.115936 13160 raft_consensus.cc:504] T 29b4025a5155470a9d4be2444635f713 P 8f65ed2870d3440097c9b56016f85408 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } }
I20260811 22:26:15.116456 13160 leader_election.cc:288] T 29b4025a5155470a9d4be2444635f713 P 8f65ed2870d3440097c9b56016f85408 [CANDIDATE]: Term 1 election: Requested vote from peers f3f6fc50d40445e3b7a35e0170a02880 (127.29.106.1:33759), d57268eda08f4b679ade15c2c4b251c9 (127.29.106.2:39563)
I20260811 22:26:15.116882 12509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b4025a5155470a9d4be2444635f713" candidate_uuid: "8f65ed2870d3440097c9b56016f85408" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3f6fc50d40445e3b7a35e0170a02880"
I20260811 22:26:15.116935 12765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b4025a5155470a9d4be2444635f713" candidate_uuid: "8f65ed2870d3440097c9b56016f85408" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d57268eda08f4b679ade15c2c4b251c9"
I20260811 22:26:15.117045 12509 raft_consensus.cc:3037] T 29b4025a5155470a9d4be2444635f713 P f3f6fc50d40445e3b7a35e0170a02880 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:26:15.117120 12765 raft_consensus.cc:3037] T 29b4025a5155470a9d4be2444635f713 P d57268eda08f4b679ade15c2c4b251c9 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:26:15.119292 12509 raft_consensus.cc:2454] T 29b4025a5155470a9d4be2444635f713 P f3f6fc50d40445e3b7a35e0170a02880 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f65ed2870d3440097c9b56016f85408 in term 1.
I20260811 22:26:15.119295 12765 raft_consensus.cc:2454] T 29b4025a5155470a9d4be2444635f713 P d57268eda08f4b679ade15c2c4b251c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f65ed2870d3440097c9b56016f85408 in term 1.
I20260811 22:26:15.119704 12890 leader_election.cc:302] T 29b4025a5155470a9d4be2444635f713 P 8f65ed2870d3440097c9b56016f85408 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8f65ed2870d3440097c9b56016f85408, d57268eda08f4b679ade15c2c4b251c9, f3f6fc50d40445e3b7a35e0170a02880; no voters:
I20260811 22:26:15.119992 13160 raft_consensus.cc:2781] T 29b4025a5155470a9d4be2444635f713 P 8f65ed2870d3440097c9b56016f85408 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:26:15.120383 13160 raft_consensus.cc:686] T 29b4025a5155470a9d4be2444635f713 P 8f65ed2870d3440097c9b56016f85408 [term 1 LEADER]: Becoming Leader. State: Replica: 8f65ed2870d3440097c9b56016f85408, State: Running, Role: LEADER
I20260811 22:26:15.120844 13160 consensus_queue.cc:237] T 29b4025a5155470a9d4be2444635f713 P 8f65ed2870d3440097c9b56016f85408 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } }
I20260811 22:26:15.124935 12227 catalog_manager.cc:5495] T 29b4025a5155470a9d4be2444635f713 P 8f65ed2870d3440097c9b56016f85408 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f65ed2870d3440097c9b56016f85408 (127.29.106.3). New cstate: current_term: 1 leader_uuid: "8f65ed2870d3440097c9b56016f85408" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } health_report { overall_health: HEALTHY } } }
W20260811 22:26:15.141675 12860 tablet.cc:2369] T 29b4025a5155470a9d4be2444635f713 P d57268eda08f4b679ade15c2c4b251c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:26:15.161955 13146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:15.151035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50922 (local address 127.29.106.3:38153)
0811 22:26:15.151253 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:15.151258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:15.151771 (+ 513us) server_negotiation.cc:407] Connection header received
0811 22:26:15.152101 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:15.152108 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:15.152252 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:15.154529 (+ 2277us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:26:15.154921 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:15.156000 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:15.157984 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:15.158514 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:15.158630 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:15.159066 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:26:15.159787 (+ 721us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:26:15.159932 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:15.161602 (+ 1670us) server_negotiation.cc:299] Negotiation successful
0811 22:26:15.161681 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260811 22:26:15.169257 13108 tablet.cc:2369] T 29b4025a5155470a9d4be2444635f713 P 8f65ed2870d3440097c9b56016f85408: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:26:15.172216 12509 raft_consensus.cc:1261] T 29b4025a5155470a9d4be2444635f713 P f3f6fc50d40445e3b7a35e0170a02880 [term 1 FOLLOWER]: Refusing update from remote peer 8f65ed2870d3440097c9b56016f85408: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:26:15.173460 12765 raft_consensus.cc:1261] T 29b4025a5155470a9d4be2444635f713 P d57268eda08f4b679ade15c2c4b251c9 [term 1 FOLLOWER]: Refusing update from remote peer 8f65ed2870d3440097c9b56016f85408: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:26:15.173240 13165 consensus_queue.cc:1035] T 29b4025a5155470a9d4be2444635f713 P 8f65ed2870d3440097c9b56016f85408 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:26:15.174240 13165 consensus_queue.cc:1035] T 29b4025a5155470a9d4be2444635f713 P 8f65ed2870d3440097c9b56016f85408 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:26:15.188648 13171 mvcc.cc:205] Tried to move back new op lower bound from 7317451469494816768 to 7317451469298380800. Current Snapshot: MvccSnapshot[applied={T|T < 7317451469494816768}]
I20260811 22:26:15.199874 12227 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:26:15.200170 12227 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:26:15.204654 12227 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0ecd4bd0e93d4f0f84a34aa8ecbe04fe: Sending DeleteTablet for 3 replicas of tablet 29b4025a5155470a9d4be2444635f713
I20260811 22:26:15.205957 12732 tablet_service.cc:1514] Processing DeleteTablet for tablet 29b4025a5155470a9d4be2444635f713 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:26:15 UTC) from {username='oryx', principal='oryx/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:41924
I20260811 22:26:15.205978 12731 tablet_service.cc:1514] Processing DeleteTablet for tablet 29b4025a5155470a9d4be2444635f713 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:26:15 UTC) from {username='oryx', principal='oryx/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:41924
I20260811 22:26:15.208734 12227 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41914:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:26:15.209363 12227 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:26:15.212091 12470 tablet_service.cc:1514] Processing DeleteTablet for tablet 29b4025a5155470a9d4be2444635f713 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:26:15 UTC) from {username='oryx', principal='oryx/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:41724
I20260811 22:26:15.212374 12968 tablet_service.cc:1514] Processing DeleteTablet for tablet 29b4025a5155470a9d4be2444635f713 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:26:15 UTC) from {username='oryx', principal='oryx/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:50906
I20260811 22:26:15.215742 13192 tablet_replica.cc:330] T 29b4025a5155470a9d4be2444635f713 P f3f6fc50d40445e3b7a35e0170a02880: stopping tablet replica
I20260811 22:26:15.216042 13193 tablet_replica.cc:330] T 29b4025a5155470a9d4be2444635f713 P 8f65ed2870d3440097c9b56016f85408: stopping tablet replica
I20260811 22:26:15.216189 13192 raft_consensus.cc:2229] T 29b4025a5155470a9d4be2444635f713 P f3f6fc50d40445e3b7a35e0170a02880 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:26:15.216574 13193 raft_consensus.cc:2229] T 29b4025a5155470a9d4be2444635f713 P 8f65ed2870d3440097c9b56016f85408 [term 1 LEADER]: Raft consensus shutting down.
I20260811 22:26:15.216683 13192 raft_consensus.cc:2258] T 29b4025a5155470a9d4be2444635f713 P f3f6fc50d40445e3b7a35e0170a02880 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:26:15.217363 13193 raft_consensus.cc:2258] T 29b4025a5155470a9d4be2444635f713 P 8f65ed2870d3440097c9b56016f85408 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:26:15.221520 12470 tablet_service.cc:1467] Processing CreateTablet for tablet c465638422494c2ba970a42cfa14b76c (DEFAULT_TABLE table=test-table [id=c8bb1172b0e9484c8025a55be36caafb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:26:15.222002 12470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c465638422494c2ba970a42cfa14b76c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:26:15.222625 13192 ts_tablet_manager.cc:1905] T 29b4025a5155470a9d4be2444635f713 P f3f6fc50d40445e3b7a35e0170a02880: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:26:15.223302 12731 tablet_service.cc:1467] Processing CreateTablet for tablet c465638422494c2ba970a42cfa14b76c (DEFAULT_TABLE table=test-table [id=c8bb1172b0e9484c8025a55be36caafb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:26:15.223765 13193 ts_tablet_manager.cc:1905] T 29b4025a5155470a9d4be2444635f713 P 8f65ed2870d3440097c9b56016f85408: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:26:15.223873 12731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c465638422494c2ba970a42cfa14b76c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:26:15.229652 13192 ts_tablet_manager.cc:1918] T 29b4025a5155470a9d4be2444635f713 P f3f6fc50d40445e3b7a35e0170a02880: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:26:15.229820 13192 log.cc:1196] T 29b4025a5155470a9d4be2444635f713 P f3f6fc50d40445e3b7a35e0170a02880: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-0/wal/wals/29b4025a5155470a9d4be2444635f713
I20260811 22:26:15.230157 12968 tablet_service.cc:1467] Processing CreateTablet for tablet c465638422494c2ba970a42cfa14b76c (DEFAULT_TABLE table=test-table [id=c8bb1172b0e9484c8025a55be36caafb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:26:15.230331 13192 ts_tablet_manager.cc:1939] T 29b4025a5155470a9d4be2444635f713 P f3f6fc50d40445e3b7a35e0170a02880: Deleting consensus metadata
I20260811 22:26:15.230597 12968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c465638422494c2ba970a42cfa14b76c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:26:15.236508 12207 catalog_manager.cc:4841] TS f3f6fc50d40445e3b7a35e0170a02880 (127.29.106.1:33759): tablet 29b4025a5155470a9d4be2444635f713 (table test-table [id=cb4939fc17c24d7fb46a4c837bda6d78]) successfully deleted
I20260811 22:26:15.237030 13190 tablet_replica.cc:330] T 29b4025a5155470a9d4be2444635f713 P d57268eda08f4b679ade15c2c4b251c9: stopping tablet replica
I20260811 22:26:15.237485 13190 raft_consensus.cc:2229] T 29b4025a5155470a9d4be2444635f713 P d57268eda08f4b679ade15c2c4b251c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:26:15.237965 13190 raft_consensus.cc:2258] T 29b4025a5155470a9d4be2444635f713 P d57268eda08f4b679ade15c2c4b251c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:26:15.241031 13149 tablet_bootstrap.cc:492] T c465638422494c2ba970a42cfa14b76c P f3f6fc50d40445e3b7a35e0170a02880: Bootstrap starting.
I20260811 22:26:15.241425 13190 ts_tablet_manager.cc:1905] T 29b4025a5155470a9d4be2444635f713 P d57268eda08f4b679ade15c2c4b251c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:26:15.242702 13149 tablet_bootstrap.cc:654] T c465638422494c2ba970a42cfa14b76c P f3f6fc50d40445e3b7a35e0170a02880: Neither blocks nor log segments found. Creating new log.
I20260811 22:26:15.245736 13151 tablet_bootstrap.cc:492] T c465638422494c2ba970a42cfa14b76c P d57268eda08f4b679ade15c2c4b251c9: Bootstrap starting.
I20260811 22:26:15.246848 13193 ts_tablet_manager.cc:1918] T 29b4025a5155470a9d4be2444635f713 P 8f65ed2870d3440097c9b56016f85408: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:26:15.247014 13193 log.cc:1196] T 29b4025a5155470a9d4be2444635f713 P 8f65ed2870d3440097c9b56016f85408: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-2/wal/wals/29b4025a5155470a9d4be2444635f713
I20260811 22:26:15.247258 13151 tablet_bootstrap.cc:654] T c465638422494c2ba970a42cfa14b76c P d57268eda08f4b679ade15c2c4b251c9: Neither blocks nor log segments found. Creating new log.
I20260811 22:26:15.247611 13193 ts_tablet_manager.cc:1939] T 29b4025a5155470a9d4be2444635f713 P 8f65ed2870d3440097c9b56016f85408: Deleting consensus metadata
I20260811 22:26:15.249357 12207 catalog_manager.cc:4841] TS 8f65ed2870d3440097c9b56016f85408 (127.29.106.3:38153): tablet 29b4025a5155470a9d4be2444635f713 (table test-table [id=cb4939fc17c24d7fb46a4c837bda6d78]) successfully deleted
I20260811 22:26:15.253185 13152 tablet_bootstrap.cc:492] T c465638422494c2ba970a42cfa14b76c P 8f65ed2870d3440097c9b56016f85408: Bootstrap starting.
I20260811 22:26:15.254753 13152 tablet_bootstrap.cc:654] T c465638422494c2ba970a42cfa14b76c P 8f65ed2870d3440097c9b56016f85408: Neither blocks nor log segments found. Creating new log.
I20260811 22:26:15.254877 13190 ts_tablet_manager.cc:1918] T 29b4025a5155470a9d4be2444635f713 P d57268eda08f4b679ade15c2c4b251c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:26:15.255014 13190 log.cc:1196] T 29b4025a5155470a9d4be2444635f713 P d57268eda08f4b679ade15c2c4b251c9: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-1/wal/wals/29b4025a5155470a9d4be2444635f713
I20260811 22:26:15.255563 13190 ts_tablet_manager.cc:1939] T 29b4025a5155470a9d4be2444635f713 P d57268eda08f4b679ade15c2c4b251c9: Deleting consensus metadata
I20260811 22:26:15.257229 12204 catalog_manager.cc:4841] TS d57268eda08f4b679ade15c2c4b251c9 (127.29.106.2:39563): tablet 29b4025a5155470a9d4be2444635f713 (table test-table [id=cb4939fc17c24d7fb46a4c837bda6d78]) successfully deleted
W20260811 22:26:15.258229 12204 catalog_manager.cc:4808] TS d57268eda08f4b679ade15c2c4b251c9 (127.29.106.2:39563): delete failed for tablet 29b4025a5155470a9d4be2444635f713 because the tablet was not found. No further retry: Not found: Tablet not found: 29b4025a5155470a9d4be2444635f713
I20260811 22:26:15.261157 13151 tablet_bootstrap.cc:492] T c465638422494c2ba970a42cfa14b76c P d57268eda08f4b679ade15c2c4b251c9: No bootstrap required, opened a new log
I20260811 22:26:15.261353 13151 ts_tablet_manager.cc:1397] T c465638422494c2ba970a42cfa14b76c P d57268eda08f4b679ade15c2c4b251c9: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.006s
I20260811 22:26:15.261978 13152 tablet_bootstrap.cc:492] T c465638422494c2ba970a42cfa14b76c P 8f65ed2870d3440097c9b56016f85408: No bootstrap required, opened a new log
I20260811 22:26:15.262131 13152 ts_tablet_manager.cc:1397] T c465638422494c2ba970a42cfa14b76c P 8f65ed2870d3440097c9b56016f85408: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.004s
I20260811 22:26:15.262419 13151 raft_consensus.cc:348] T c465638422494c2ba970a42cfa14b76c P d57268eda08f4b679ade15c2c4b251c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } }
I20260811 22:26:15.262718 13151 raft_consensus.cc:374] T c465638422494c2ba970a42cfa14b76c P d57268eda08f4b679ade15c2c4b251c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:26:15.262863 13151 raft_consensus.cc:729] T c465638422494c2ba970a42cfa14b76c P d57268eda08f4b679ade15c2c4b251c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d57268eda08f4b679ade15c2c4b251c9, State: Initialized, Role: FOLLOWER
I20260811 22:26:15.262892 13152 raft_consensus.cc:348] T c465638422494c2ba970a42cfa14b76c P 8f65ed2870d3440097c9b56016f85408 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } }
I20260811 22:26:15.263024 13149 tablet_bootstrap.cc:492] T c465638422494c2ba970a42cfa14b76c P f3f6fc50d40445e3b7a35e0170a02880: No bootstrap required, opened a new log
I20260811 22:26:15.263108 13152 raft_consensus.cc:374] T c465638422494c2ba970a42cfa14b76c P 8f65ed2870d3440097c9b56016f85408 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:26:15.263175 13149 ts_tablet_manager.cc:1397] T c465638422494c2ba970a42cfa14b76c P f3f6fc50d40445e3b7a35e0170a02880: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.013s
I20260811 22:26:15.263214 13152 raft_consensus.cc:729] T c465638422494c2ba970a42cfa14b76c P 8f65ed2870d3440097c9b56016f85408 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f65ed2870d3440097c9b56016f85408, State: Initialized, Role: FOLLOWER
I20260811 22:26:15.263430 13151 consensus_queue.cc:260] T c465638422494c2ba970a42cfa14b76c P d57268eda08f4b679ade15c2c4b251c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } }
I20260811 22:26:15.263553 13152 consensus_queue.cc:260] T c465638422494c2ba970a42cfa14b76c P 8f65ed2870d3440097c9b56016f85408 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } }
I20260811 22:26:15.264099 13151 ts_tablet_manager.cc:1428] T c465638422494c2ba970a42cfa14b76c P d57268eda08f4b679ade15c2c4b251c9: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260811 22:26:15.264166 13152 ts_tablet_manager.cc:1428] T c465638422494c2ba970a42cfa14b76c P 8f65ed2870d3440097c9b56016f85408: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260811 22:26:15.265285 13149 raft_consensus.cc:348] T c465638422494c2ba970a42cfa14b76c P f3f6fc50d40445e3b7a35e0170a02880 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } }
I20260811 22:26:15.265501 13149 raft_consensus.cc:374] T c465638422494c2ba970a42cfa14b76c P f3f6fc50d40445e3b7a35e0170a02880 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:26:15.265594 13149 raft_consensus.cc:729] T c465638422494c2ba970a42cfa14b76c P f3f6fc50d40445e3b7a35e0170a02880 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3f6fc50d40445e3b7a35e0170a02880, State: Initialized, Role: FOLLOWER
I20260811 22:26:15.265841 13149 consensus_queue.cc:260] T c465638422494c2ba970a42cfa14b76c P f3f6fc50d40445e3b7a35e0170a02880 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } }
I20260811 22:26:15.268752 13149 ts_tablet_manager.cc:1428] T c465638422494c2ba970a42cfa14b76c P f3f6fc50d40445e3b7a35e0170a02880: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260811 22:26:15.269460 13158 raft_consensus.cc:482] T c465638422494c2ba970a42cfa14b76c P d57268eda08f4b679ade15c2c4b251c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:26:15.269676 13158 raft_consensus.cc:504] T c465638422494c2ba970a42cfa14b76c P d57268eda08f4b679ade15c2c4b251c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } }
I20260811 22:26:15.271065 13158 leader_election.cc:288] T c465638422494c2ba970a42cfa14b76c P d57268eda08f4b679ade15c2c4b251c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3f6fc50d40445e3b7a35e0170a02880 (127.29.106.1:33759), 8f65ed2870d3440097c9b56016f85408 (127.29.106.3:38153)
I20260811 22:26:15.279546 13139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:15.271672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:40143 (local address 127.29.106.1:33759)
0811 22:26:15.271843 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:15.271849 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:15.272060 (+ 211us) server_negotiation.cc:407] Connection header received
0811 22:26:15.272326 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:15.272331 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:15.272401 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:15.272462 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:26:15.273515 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:15.274742 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:15.277416 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:15.278557 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:15.278650 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:15.278699 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:15.279145 (+ 446us) server_negotiation.cc:299] Negotiation successful
0811 22:26:15.279290 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260811 22:26:15.279589 13197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:15.271310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:33759 (local address 127.29.106.2:40143)
0811 22:26:15.271972 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:15.271990 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:15.272161 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:15.272872 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:15.272877 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:15.272906 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:26:15.273314 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:15.273326 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:15.274982 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:15.274988 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:15.277241 (+ 2253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:15.277258 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:15.278869 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:15.278873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:15.278996 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:15.279017 (+ 21us) client_negotiation.cc:769] Sending connection context
0811 22:26:15.279097 (+ 80us) client_negotiation.cc:240] Negotiation successful
0811 22:26:15.279225 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":430,"threads_started":1}
I20260811 22:26:15.280314 12509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c465638422494c2ba970a42cfa14b76c" candidate_uuid: "d57268eda08f4b679ade15c2c4b251c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3f6fc50d40445e3b7a35e0170a02880" is_pre_election: true
I20260811 22:26:15.280515 12509 raft_consensus.cc:2454] T c465638422494c2ba970a42cfa14b76c P f3f6fc50d40445e3b7a35e0170a02880 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d57268eda08f4b679ade15c2c4b251c9 in term 0.
I20260811 22:26:15.280941 12650 leader_election.cc:302] T c465638422494c2ba970a42cfa14b76c P d57268eda08f4b679ade15c2c4b251c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d57268eda08f4b679ade15c2c4b251c9, f3f6fc50d40445e3b7a35e0170a02880; no voters:
I20260811 22:26:15.281330 13158 raft_consensus.cc:2781] T c465638422494c2ba970a42cfa14b76c P d57268eda08f4b679ade15c2c4b251c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:26:15.281456 13158 raft_consensus.cc:482] T c465638422494c2ba970a42cfa14b76c P d57268eda08f4b679ade15c2c4b251c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:26:15.281546 13158 raft_consensus.cc:3037] T c465638422494c2ba970a42cfa14b76c P d57268eda08f4b679ade15c2c4b251c9 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:26:15.283090 13146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:15.272402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:40569 (local address 127.29.106.3:38153)
0811 22:26:15.272742 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:15.272748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:15.276082 (+ 3334us) server_negotiation.cc:407] Connection header received
0811 22:26:15.276260 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:15.276266 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:15.276356 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:15.276452 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:26:15.277103 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:15.278219 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:15.280733 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:15.281861 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:15.282031 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:15.282103 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:15.282653 (+ 550us) server_negotiation.cc:299] Negotiation successful
0811 22:26:15.282790 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213}
I20260811 22:26:15.283723 13199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:15.272103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.3:38153 (local address 127.29.106.2:40569)
0811 22:26:15.275961 (+ 3858us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:15.275983 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:15.276125 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:15.276511 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:15.276515 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:15.276541 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:26:15.276930 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:15.276943 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:15.278356 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:15.278362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:15.280572 (+ 2210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:15.280587 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:15.282264 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:15.282270 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:15.282451 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:15.282483 (+ 32us) client_negotiation.cc:769] Sending connection context
0811 22:26:15.283242 (+ 759us) client_negotiation.cc:240] Negotiation successful
0811 22:26:15.283399 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3709,"thread_start_us":3528,"threads_started":1}
I20260811 22:26:15.284446 13158 raft_consensus.cc:504] T c465638422494c2ba970a42cfa14b76c P d57268eda08f4b679ade15c2c4b251c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } }
I20260811 22:26:15.284869 13158 leader_election.cc:288] T c465638422494c2ba970a42cfa14b76c P d57268eda08f4b679ade15c2c4b251c9 [CANDIDATE]: Term 1 election: Requested vote from peers f3f6fc50d40445e3b7a35e0170a02880 (127.29.106.1:33759), 8f65ed2870d3440097c9b56016f85408 (127.29.106.3:38153)
I20260811 22:26:15.288101 12509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c465638422494c2ba970a42cfa14b76c" candidate_uuid: "d57268eda08f4b679ade15c2c4b251c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3f6fc50d40445e3b7a35e0170a02880"
I20260811 22:26:15.288314 12509 raft_consensus.cc:3037] T c465638422494c2ba970a42cfa14b76c P f3f6fc50d40445e3b7a35e0170a02880 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:26:15.288448 13015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c465638422494c2ba970a42cfa14b76c" candidate_uuid: "d57268eda08f4b679ade15c2c4b251c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f65ed2870d3440097c9b56016f85408" is_pre_election: true
I20260811 22:26:15.288892 13015 raft_consensus.cc:2454] T c465638422494c2ba970a42cfa14b76c P 8f65ed2870d3440097c9b56016f85408 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d57268eda08f4b679ade15c2c4b251c9 in term 0.
I20260811 22:26:15.289769 13015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c465638422494c2ba970a42cfa14b76c" candidate_uuid: "d57268eda08f4b679ade15c2c4b251c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f65ed2870d3440097c9b56016f85408"
I20260811 22:26:15.289948 13015 raft_consensus.cc:3037] T c465638422494c2ba970a42cfa14b76c P 8f65ed2870d3440097c9b56016f85408 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:26:15.291141 12509 raft_consensus.cc:2454] T c465638422494c2ba970a42cfa14b76c P f3f6fc50d40445e3b7a35e0170a02880 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d57268eda08f4b679ade15c2c4b251c9 in term 1.
I20260811 22:26:15.292596 13015 raft_consensus.cc:2454] T c465638422494c2ba970a42cfa14b76c P 8f65ed2870d3440097c9b56016f85408 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d57268eda08f4b679ade15c2c4b251c9 in term 1.
I20260811 22:26:15.292948 12650 leader_election.cc:302] T c465638422494c2ba970a42cfa14b76c P d57268eda08f4b679ade15c2c4b251c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d57268eda08f4b679ade15c2c4b251c9, f3f6fc50d40445e3b7a35e0170a02880; no voters:
I20260811 22:26:15.293591 13158 raft_consensus.cc:2781] T c465638422494c2ba970a42cfa14b76c P d57268eda08f4b679ade15c2c4b251c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:26:15.294063 13158 raft_consensus.cc:686] T c465638422494c2ba970a42cfa14b76c P d57268eda08f4b679ade15c2c4b251c9 [term 1 LEADER]: Becoming Leader. State: Replica: d57268eda08f4b679ade15c2c4b251c9, State: Running, Role: LEADER
I20260811 22:26:15.294435 13158 consensus_queue.cc:237] T c465638422494c2ba970a42cfa14b76c P d57268eda08f4b679ade15c2c4b251c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } }
I20260811 22:26:15.297677 12225 catalog_manager.cc:5495] T c465638422494c2ba970a42cfa14b76c P d57268eda08f4b679ade15c2c4b251c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to d57268eda08f4b679ade15c2c4b251c9 (127.29.106.2). New cstate: current_term: 1 leader_uuid: "d57268eda08f4b679ade15c2c4b251c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } health_report { overall_health: UNKNOWN } } }
Aug 11 22:26:15 dist-test-slave-gjrz krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487172, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.106.62@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM
I20260811 22:26:15.343218 12316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:15.329612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41924 (local address 127.29.106.62:35049)
0811 22:26:15.329761 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:15.329766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:15.330359 (+ 593us) server_negotiation.cc:407] Connection header received
0811 22:26:15.330522 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:15.330527 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:15.330589 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:15.330667 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:15.331839 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:15.332577 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:15.333778 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:15.334194 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:15.334299 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:15.337731 (+ 3432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:15.337736 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:15.337739 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:15.337768 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:15.340294 (+ 2526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:15.341142 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:15.341146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:15.341148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:15.341263 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:15.341667 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:15.341670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:15.341672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:15.341957 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:15.342104 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:15.342619 (+ 515us) server_negotiation.cc:299] Negotiation successful
0811 22:26:15.342918 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":112,"server-negotiator.queue_time_us":50}
I20260811 22:26:15.343664 13213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:15.329720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:35049 (local address 127.0.0.1:41924)
0811 22:26:15.330259 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:15.330277 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:15.330418 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:15.330703 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:15.330707 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:15.331438 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:15.331706 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:15.331716 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:15.332695 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:15.332700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:15.333649 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:15.333660 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:15.334359 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:15.334366 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:15.334508 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:15.335160 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:15.335177 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:15.337568 (+ 2391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:26:15.340429 (+ 2861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:15.340442 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:15.340451 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:15.341024 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:15.341364 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:15.341370 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:15.341374 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:15.341566 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:15.342114 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:26:15.342122 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:26:15.342319 (+ 197us) client_negotiation.cc:769] Sending connection context
0811 22:26:15.342509 (+ 190us) client_negotiation.cc:240] Negotiation successful
0811 22:26:15.342790 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":318,"threads_started":1}
I20260811 22:26:15.348640 12225 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:41924:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260811 22:26:15.349915 12225 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:26:15.356315 12470 tablet_service.cc:1467] Processing CreateTablet for tablet 5b60814b7cc44916a72652b71867a1ca (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d7d6bf10bce84d44a13810820bf0e9b3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 22:26:15.356621 12731 tablet_service.cc:1467] Processing CreateTablet for tablet 5b60814b7cc44916a72652b71867a1ca (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d7d6bf10bce84d44a13810820bf0e9b3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 22:26:15.356647 12968 tablet_service.cc:1467] Processing CreateTablet for tablet 5b60814b7cc44916a72652b71867a1ca (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d7d6bf10bce84d44a13810820bf0e9b3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 22:26:15.356909 12470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b60814b7cc44916a72652b71867a1ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:26:15.357116 12731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b60814b7cc44916a72652b71867a1ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:26:15.357215 12968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b60814b7cc44916a72652b71867a1ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:26:15.364145 13149 tablet_bootstrap.cc:492] T 5b60814b7cc44916a72652b71867a1ca P f3f6fc50d40445e3b7a35e0170a02880: Bootstrap starting.
I20260811 22:26:15.364203 13151 tablet_bootstrap.cc:492] T 5b60814b7cc44916a72652b71867a1ca P d57268eda08f4b679ade15c2c4b251c9: Bootstrap starting.
I20260811 22:26:15.364801 13152 tablet_bootstrap.cc:492] T 5b60814b7cc44916a72652b71867a1ca P 8f65ed2870d3440097c9b56016f85408: Bootstrap starting.
I20260811 22:26:15.365759 13151 tablet_bootstrap.cc:654] T 5b60814b7cc44916a72652b71867a1ca P d57268eda08f4b679ade15c2c4b251c9: Neither blocks nor log segments found. Creating new log.
I20260811 22:26:15.365756 13149 tablet_bootstrap.cc:654] T 5b60814b7cc44916a72652b71867a1ca P f3f6fc50d40445e3b7a35e0170a02880: Neither blocks nor log segments found. Creating new log.
I20260811 22:26:15.366496 13152 tablet_bootstrap.cc:654] T 5b60814b7cc44916a72652b71867a1ca P 8f65ed2870d3440097c9b56016f85408: Neither blocks nor log segments found. Creating new log.
I20260811 22:26:15.376332 13152 tablet_bootstrap.cc:492] T 5b60814b7cc44916a72652b71867a1ca P 8f65ed2870d3440097c9b56016f85408: No bootstrap required, opened a new log
I20260811 22:26:15.376502 13152 ts_tablet_manager.cc:1397] T 5b60814b7cc44916a72652b71867a1ca P 8f65ed2870d3440097c9b56016f85408: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20260811 22:26:15.376745 13151 tablet_bootstrap.cc:492] T 5b60814b7cc44916a72652b71867a1ca P d57268eda08f4b679ade15c2c4b251c9: No bootstrap required, opened a new log
I20260811 22:26:15.376952 13151 ts_tablet_manager.cc:1397] T 5b60814b7cc44916a72652b71867a1ca P d57268eda08f4b679ade15c2c4b251c9: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.011s
I20260811 22:26:15.377308 13152 raft_consensus.cc:348] T 5b60814b7cc44916a72652b71867a1ca P 8f65ed2870d3440097c9b56016f85408 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } }
I20260811 22:26:15.377573 13152 raft_consensus.cc:374] T 5b60814b7cc44916a72652b71867a1ca P 8f65ed2870d3440097c9b56016f85408 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:26:15.377664 13152 raft_consensus.cc:729] T 5b60814b7cc44916a72652b71867a1ca P 8f65ed2870d3440097c9b56016f85408 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f65ed2870d3440097c9b56016f85408, State: Initialized, Role: FOLLOWER
I20260811 22:26:15.377910 13151 raft_consensus.cc:348] T 5b60814b7cc44916a72652b71867a1ca P d57268eda08f4b679ade15c2c4b251c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } }
I20260811 22:26:15.377949 13152 consensus_queue.cc:260] T 5b60814b7cc44916a72652b71867a1ca P 8f65ed2870d3440097c9b56016f85408 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } }
I20260811 22:26:15.378253 13151 raft_consensus.cc:374] T 5b60814b7cc44916a72652b71867a1ca P d57268eda08f4b679ade15c2c4b251c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:26:15.378377 13151 raft_consensus.cc:729] T 5b60814b7cc44916a72652b71867a1ca P d57268eda08f4b679ade15c2c4b251c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d57268eda08f4b679ade15c2c4b251c9, State: Initialized, Role: FOLLOWER
I20260811 22:26:15.378696 13149 tablet_bootstrap.cc:492] T 5b60814b7cc44916a72652b71867a1ca P f3f6fc50d40445e3b7a35e0170a02880: No bootstrap required, opened a new log
I20260811 22:26:15.378433 13183 tablet_replica.cc:439] T 5b60814b7cc44916a72652b71867a1ca P 8f65ed2870d3440097c9b56016f85408: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } } }
I20260811 22:26:15.378862 13183 tablet_replica.cc:442] T 5b60814b7cc44916a72652b71867a1ca P 8f65ed2870d3440097c9b56016f85408: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 22:26:15.378870 13149 ts_tablet_manager.cc:1397] T 5b60814b7cc44916a72652b71867a1ca P f3f6fc50d40445e3b7a35e0170a02880: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.007s
I20260811 22:26:15.378734 13151 consensus_queue.cc:260] T 5b60814b7cc44916a72652b71867a1ca P d57268eda08f4b679ade15c2c4b251c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } }
I20260811 22:26:15.379530 13158 tablet_replica.cc:439] T 5b60814b7cc44916a72652b71867a1ca P d57268eda08f4b679ade15c2c4b251c9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } } }
I20260811 22:26:15.379817 13158 tablet_replica.cc:442] T 5b60814b7cc44916a72652b71867a1ca P d57268eda08f4b679ade15c2c4b251c9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 22:26:15.380264 13152 ts_tablet_manager.cc:1428] T 5b60814b7cc44916a72652b71867a1ca P 8f65ed2870d3440097c9b56016f85408: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260811 22:26:15.380533 13149 raft_consensus.cc:348] T 5b60814b7cc44916a72652b71867a1ca P f3f6fc50d40445e3b7a35e0170a02880 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } }
I20260811 22:26:15.380759 13149 raft_consensus.cc:374] T 5b60814b7cc44916a72652b71867a1ca P f3f6fc50d40445e3b7a35e0170a02880 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:26:15.380832 13149 raft_consensus.cc:729] T 5b60814b7cc44916a72652b71867a1ca P f3f6fc50d40445e3b7a35e0170a02880 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3f6fc50d40445e3b7a35e0170a02880, State: Initialized, Role: FOLLOWER
I20260811 22:26:15.381088 13149 consensus_queue.cc:260] T 5b60814b7cc44916a72652b71867a1ca P f3f6fc50d40445e3b7a35e0170a02880 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } }
I20260811 22:26:15.381675 13149 ts_tablet_manager.cc:1428] T 5b60814b7cc44916a72652b71867a1ca P f3f6fc50d40445e3b7a35e0170a02880: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260811 22:26:15.381577 13159 tablet_replica.cc:439] T 5b60814b7cc44916a72652b71867a1ca P f3f6fc50d40445e3b7a35e0170a02880: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } } }
I20260811 22:26:15.381948 13159 tablet_replica.cc:442] T 5b60814b7cc44916a72652b71867a1ca P f3f6fc50d40445e3b7a35e0170a02880: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 22:26:15.395557 13151 ts_tablet_manager.cc:1428] T 5b60814b7cc44916a72652b71867a1ca P d57268eda08f4b679ade15c2c4b251c9: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20260811 22:26:15.487051 13158 raft_consensus.cc:482] T 5b60814b7cc44916a72652b71867a1ca P d57268eda08f4b679ade15c2c4b251c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:26:15.487282 13158 raft_consensus.cc:504] T 5b60814b7cc44916a72652b71867a1ca P d57268eda08f4b679ade15c2c4b251c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } }
I20260811 22:26:15.487983 13158 leader_election.cc:288] T 5b60814b7cc44916a72652b71867a1ca P d57268eda08f4b679ade15c2c4b251c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3f6fc50d40445e3b7a35e0170a02880 (127.29.106.1:33759), 8f65ed2870d3440097c9b56016f85408 (127.29.106.3:38153)
I20260811 22:26:15.488596 12509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b60814b7cc44916a72652b71867a1ca" candidate_uuid: "d57268eda08f4b679ade15c2c4b251c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3f6fc50d40445e3b7a35e0170a02880" is_pre_election: true
I20260811 22:26:15.488595 13015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b60814b7cc44916a72652b71867a1ca" candidate_uuid: "d57268eda08f4b679ade15c2c4b251c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f65ed2870d3440097c9b56016f85408" is_pre_election: true
I20260811 22:26:15.488880 12509 raft_consensus.cc:2454] T 5b60814b7cc44916a72652b71867a1ca P f3f6fc50d40445e3b7a35e0170a02880 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d57268eda08f4b679ade15c2c4b251c9 in term 0.
I20260811 22:26:15.488901 13015 raft_consensus.cc:2454] T 5b60814b7cc44916a72652b71867a1ca P 8f65ed2870d3440097c9b56016f85408 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d57268eda08f4b679ade15c2c4b251c9 in term 0.
I20260811 22:26:15.489326 12650 leader_election.cc:302] T 5b60814b7cc44916a72652b71867a1ca P d57268eda08f4b679ade15c2c4b251c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d57268eda08f4b679ade15c2c4b251c9, f3f6fc50d40445e3b7a35e0170a02880; no voters:
I20260811 22:26:15.489594 13158 raft_consensus.cc:2781] T 5b60814b7cc44916a72652b71867a1ca P d57268eda08f4b679ade15c2c4b251c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:26:15.489705 13158 raft_consensus.cc:482] T 5b60814b7cc44916a72652b71867a1ca P d57268eda08f4b679ade15c2c4b251c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:26:15.489812 13158 raft_consensus.cc:3037] T 5b60814b7cc44916a72652b71867a1ca P d57268eda08f4b679ade15c2c4b251c9 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:26:15.492254 13158 raft_consensus.cc:504] T 5b60814b7cc44916a72652b71867a1ca P d57268eda08f4b679ade15c2c4b251c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } }
I20260811 22:26:15.492784 13158 leader_election.cc:288] T 5b60814b7cc44916a72652b71867a1ca P d57268eda08f4b679ade15c2c4b251c9 [CANDIDATE]: Term 1 election: Requested vote from peers f3f6fc50d40445e3b7a35e0170a02880 (127.29.106.1:33759), 8f65ed2870d3440097c9b56016f85408 (127.29.106.3:38153)
I20260811 22:26:15.494433 12509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b60814b7cc44916a72652b71867a1ca" candidate_uuid: "d57268eda08f4b679ade15c2c4b251c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3f6fc50d40445e3b7a35e0170a02880"
I20260811 22:26:15.494664 12509 raft_consensus.cc:3037] T 5b60814b7cc44916a72652b71867a1ca P f3f6fc50d40445e3b7a35e0170a02880 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:26:15.494580 13015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b60814b7cc44916a72652b71867a1ca" candidate_uuid: "d57268eda08f4b679ade15c2c4b251c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f65ed2870d3440097c9b56016f85408"
I20260811 22:26:15.494887 13015 raft_consensus.cc:3037] T 5b60814b7cc44916a72652b71867a1ca P 8f65ed2870d3440097c9b56016f85408 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:26:15.497737 12509 raft_consensus.cc:2454] T 5b60814b7cc44916a72652b71867a1ca P f3f6fc50d40445e3b7a35e0170a02880 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d57268eda08f4b679ade15c2c4b251c9 in term 1.
I20260811 22:26:15.498165 12650 leader_election.cc:302] T 5b60814b7cc44916a72652b71867a1ca P d57268eda08f4b679ade15c2c4b251c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d57268eda08f4b679ade15c2c4b251c9, f3f6fc50d40445e3b7a35e0170a02880; no voters:
I20260811 22:26:15.498518 13158 raft_consensus.cc:2781] T 5b60814b7cc44916a72652b71867a1ca P d57268eda08f4b679ade15c2c4b251c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:26:15.499022 13158 raft_consensus.cc:686] T 5b60814b7cc44916a72652b71867a1ca P d57268eda08f4b679ade15c2c4b251c9 [term 1 LEADER]: Becoming Leader. State: Replica: d57268eda08f4b679ade15c2c4b251c9, State: Running, Role: LEADER
I20260811 22:26:15.499258 13158 consensus_queue.cc:237] T 5b60814b7cc44916a72652b71867a1ca P d57268eda08f4b679ade15c2c4b251c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } }
I20260811 22:26:15.499490 13015 raft_consensus.cc:2454] T 5b60814b7cc44916a72652b71867a1ca P 8f65ed2870d3440097c9b56016f85408 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d57268eda08f4b679ade15c2c4b251c9 in term 1.
I20260811 22:26:15.507917 13200 tablet_replica.cc:439] T 5b60814b7cc44916a72652b71867a1ca P d57268eda08f4b679ade15c2c4b251c9: TxnStatusTablet state changed. Reason: New leader d57268eda08f4b679ade15c2c4b251c9. Latest consensus state: current_term: 1 leader_uuid: "d57268eda08f4b679ade15c2c4b251c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } } }
I20260811 22:26:15.508394 13200 tablet_replica.cc:442] T 5b60814b7cc44916a72652b71867a1ca P d57268eda08f4b679ade15c2c4b251c9: This TxnStatusTablet replica's current role is: LEADER
I20260811 22:26:15.509158 13227 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260811 22:26:15.509415 13227 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260811 22:26:15.509182 12225 catalog_manager.cc:5495] T 5b60814b7cc44916a72652b71867a1ca P d57268eda08f4b679ade15c2c4b251c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to d57268eda08f4b679ade15c2c4b251c9 (127.29.106.2). New cstate: current_term: 1 leader_uuid: "d57268eda08f4b679ade15c2c4b251c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } health_report { overall_health: UNKNOWN } } }
I20260811 22:26:15.534096 13213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:15.526641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.2:39563 (local address 127.0.0.1:41940)
0811 22:26:15.526893 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:15.526913 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:15.527061 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:15.527634 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:15.527639 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:15.527653 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 22:26:15.527857 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:15.527900 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:15.529421 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:15.529427 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:15.531000 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:15.531016 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:15.532165 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:15.532171 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:15.532352 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:15.532427 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 22:26:15.533512 (+ 1085us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 22:26:15.533535 (+ 23us) client_negotiation.cc:769] Sending connection context
0811 22:26:15.533622 (+ 87us) client_negotiation.cc:240] Negotiation successful
0811 22:26:15.533786 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260811 22:26:15.534103 13144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:15.526750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41940 (local address 127.29.106.2:39563)
0811 22:26:15.526932 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:15.526938 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:15.526962 (+ 24us) server_negotiation.cc:407] Connection header received
0811 22:26:15.527323 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:15.527328 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:15.527439 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:15.527599 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:26:15.528021 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:15.529245 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:15.531171 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:15.531877 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:15.532002 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:15.532675 (+ 673us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:26:15.533381 (+ 706us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:26:15.533548 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:15.533664 (+ 116us) server_negotiation.cc:299] Negotiation successful
0811 22:26:15.533755 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260811 22:26:15.538910 12509 raft_consensus.cc:1261] T 5b60814b7cc44916a72652b71867a1ca P f3f6fc50d40445e3b7a35e0170a02880 [term 1 FOLLOWER]: Refusing update from remote peer d57268eda08f4b679ade15c2c4b251c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:26:15.539115 13015 raft_consensus.cc:1261] T 5b60814b7cc44916a72652b71867a1ca P 8f65ed2870d3440097c9b56016f85408 [term 1 FOLLOWER]: Refusing update from remote peer d57268eda08f4b679ade15c2c4b251c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:26:15.540031 13158 consensus_queue.cc:1035] T 5b60814b7cc44916a72652b71867a1ca P d57268eda08f4b679ade15c2c4b251c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:26:15.540700 13158 consensus_queue.cc:1035] T 5b60814b7cc44916a72652b71867a1ca P d57268eda08f4b679ade15c2c4b251c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:26:15.552268 13159 tablet_replica.cc:439] T 5b60814b7cc44916a72652b71867a1ca P f3f6fc50d40445e3b7a35e0170a02880: TxnStatusTablet state changed. Reason: New leader d57268eda08f4b679ade15c2c4b251c9. Latest consensus state: current_term: 1 leader_uuid: "d57268eda08f4b679ade15c2c4b251c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } } }
I20260811 22:26:15.552552 13159 tablet_replica.cc:442] T 5b60814b7cc44916a72652b71867a1ca P f3f6fc50d40445e3b7a35e0170a02880: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 22:26:15.553184 13183 tablet_replica.cc:439] T 5b60814b7cc44916a72652b71867a1ca P 8f65ed2870d3440097c9b56016f85408: TxnStatusTablet state changed. Reason: New leader d57268eda08f4b679ade15c2c4b251c9. Latest consensus state: current_term: 1 leader_uuid: "d57268eda08f4b679ade15c2c4b251c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } } }
I20260811 22:26:15.553491 13183 tablet_replica.cc:442] T 5b60814b7cc44916a72652b71867a1ca P 8f65ed2870d3440097c9b56016f85408: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 22:26:15.556545 13200 tablet_replica.cc:439] T 5b60814b7cc44916a72652b71867a1ca P d57268eda08f4b679ade15c2c4b251c9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d57268eda08f4b679ade15c2c4b251c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } } }
I20260811 22:26:15.556809 13200 tablet_replica.cc:442] T 5b60814b7cc44916a72652b71867a1ca P d57268eda08f4b679ade15c2c4b251c9: This TxnStatusTablet replica's current role is: LEADER
I20260811 22:26:15.557008 13174 mvcc.cc:205] Tried to move back new op lower bound from 7317451470999171072 to 7317451470848688128. Current Snapshot: MvccSnapshot[applied={T|T < 7317451470999171072 or (T in {7317451470999171072})}]
I20260811 22:26:15.559374 13168 mvcc.cc:205] Tried to move back new op lower bound from 7317451470999171072 to 7317451470848688128. Current Snapshot: MvccSnapshot[applied={T|T < 7317451470999171072}]
I20260811 22:26:15.559672 13183 tablet_replica.cc:439] T 5b60814b7cc44916a72652b71867a1ca P 8f65ed2870d3440097c9b56016f85408: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d57268eda08f4b679ade15c2c4b251c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } } }
I20260811 22:26:15.559907 13183 tablet_replica.cc:442] T 5b60814b7cc44916a72652b71867a1ca P 8f65ed2870d3440097c9b56016f85408: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 22:26:15.563181 13158 tablet_replica.cc:439] T 5b60814b7cc44916a72652b71867a1ca P d57268eda08f4b679ade15c2c4b251c9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d57268eda08f4b679ade15c2c4b251c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } } }
I20260811 22:26:15.563494 13158 tablet_replica.cc:442] T 5b60814b7cc44916a72652b71867a1ca P d57268eda08f4b679ade15c2c4b251c9: This TxnStatusTablet replica's current role is: LEADER
I20260811 22:26:15.567147 13159 tablet_replica.cc:439] T 5b60814b7cc44916a72652b71867a1ca P f3f6fc50d40445e3b7a35e0170a02880: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d57268eda08f4b679ade15c2c4b251c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } } }
I20260811 22:26:15.567399 13159 tablet_replica.cc:442] T 5b60814b7cc44916a72652b71867a1ca P f3f6fc50d40445e3b7a35e0170a02880: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 22:26:15.572364 13144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:15.565830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41942 (local address 127.29.106.2:39563)
0811 22:26:15.566142 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:15.566148 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:15.566166 (+ 18us) server_negotiation.cc:407] Connection header received
0811 22:26:15.566224 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:15.566228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:15.566299 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:15.566386 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:26:15.566966 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:15.568120 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:15.570632 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:15.571074 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:15.571176 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:15.571675 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:26:15.571769 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:26:15.571839 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:15.571982 (+ 143us) server_negotiation.cc:299] Negotiation successful
0811 22:26:15.572056 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
I20260811 22:26:15.587014 13144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:15.581700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:33005 (local address 127.29.106.2:39563)
0811 22:26:15.581918 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:15.581925 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:15.582430 (+ 505us) server_negotiation.cc:407] Connection header received
0811 22:26:15.582590 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:15.582595 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:15.582656 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:15.582727 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:26:15.583243 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:15.583983 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:15.585392 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:15.585815 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:15.585918 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:15.586309 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:26:15.586420 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:26:15.586488 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:15.586625 (+ 137us) server_negotiation.cc:299] Negotiation successful
0811 22:26:15.586690 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":55}
I20260811 22:26:15.587219 13241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:15.581518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.2:39563 (local address 127.29.106.2:33005)
0811 22:26:15.582338 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:15.582353 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:15.582482 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:15.582750 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:15.582755 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:15.582766 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 22:26:15.583116 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:15.583125 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:15.584096 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:15.584102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:15.585269 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:15.585280 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:15.585984 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:15.585990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:15.586110 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:15.586162 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 22:26:15.586508 (+ 346us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 22:26:15.586519 (+ 11us) client_negotiation.cc:769] Sending connection context
0811 22:26:15.586602 (+ 83us) client_negotiation.cc:240] Negotiation successful
0811 22:26:15.586731 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"mutex_wait_us":2,"thread_start_us":551,"threads_started":1}
I20260811 22:26:15.596222 12507 raft_consensus.cc:1261] T c465638422494c2ba970a42cfa14b76c P f3f6fc50d40445e3b7a35e0170a02880 [term 1 FOLLOWER]: Refusing update from remote peer d57268eda08f4b679ade15c2c4b251c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:26:15.596704 13013 raft_consensus.cc:1261] T c465638422494c2ba970a42cfa14b76c P 8f65ed2870d3440097c9b56016f85408 [term 1 FOLLOWER]: Refusing update from remote peer d57268eda08f4b679ade15c2c4b251c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:26:15.596953 13200 consensus_queue.cc:1035] T c465638422494c2ba970a42cfa14b76c P d57268eda08f4b679ade15c2c4b251c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:26:15.599678 13158 consensus_queue.cc:1035] T c465638422494c2ba970a42cfa14b76c P d57268eda08f4b679ade15c2c4b251c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:26:15.684064 12225 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:26:15.684319 12225 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:26:15.687635 12225 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0ecd4bd0e93d4f0f84a34aa8ecbe04fe: Sending DeleteTablet for 3 replicas of tablet c465638422494c2ba970a42cfa14b76c
I20260811 22:26:15.688516 12470 tablet_service.cc:1514] Processing DeleteTablet for tablet c465638422494c2ba970a42cfa14b76c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:26:15 UTC) from {username='oryx', principal='oryx/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:41724
I20260811 22:26:15.688685 12968 tablet_service.cc:1514] Processing DeleteTablet for tablet c465638422494c2ba970a42cfa14b76c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:26:15 UTC) from {username='oryx', principal='oryx/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:50906
I20260811 22:26:15.688795 13192 tablet_replica.cc:330] T c465638422494c2ba970a42cfa14b76c P f3f6fc50d40445e3b7a35e0170a02880: stopping tablet replica
I20260811 22:26:15.688951 13193 tablet_replica.cc:330] T c465638422494c2ba970a42cfa14b76c P 8f65ed2870d3440097c9b56016f85408: stopping tablet replica
I20260811 22:26:15.688999 13192 raft_consensus.cc:2229] T c465638422494c2ba970a42cfa14b76c P f3f6fc50d40445e3b7a35e0170a02880 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:26:15.689165 13193 raft_consensus.cc:2229] T c465638422494c2ba970a42cfa14b76c P 8f65ed2870d3440097c9b56016f85408 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:26:15.689222 13192 raft_consensus.cc:2258] T c465638422494c2ba970a42cfa14b76c P f3f6fc50d40445e3b7a35e0170a02880 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:26:15.689359 13193 raft_consensus.cc:2258] T c465638422494c2ba970a42cfa14b76c P 8f65ed2870d3440097c9b56016f85408 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:26:15.689352 12731 tablet_service.cc:1514] Processing DeleteTablet for tablet c465638422494c2ba970a42cfa14b76c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:26:15 UTC) from {username='oryx', principal='oryx/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:41924
I20260811 22:26:15.690680 13193 ts_tablet_manager.cc:1905] T c465638422494c2ba970a42cfa14b76c P 8f65ed2870d3440097c9b56016f85408: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:26:15.690909 13192 ts_tablet_manager.cc:1905] T c465638422494c2ba970a42cfa14b76c P f3f6fc50d40445e3b7a35e0170a02880: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:26:15.696074 13190 tablet_replica.cc:330] T c465638422494c2ba970a42cfa14b76c P d57268eda08f4b679ade15c2c4b251c9: stopping tablet replica
I20260811 22:26:15.696271 13190 raft_consensus.cc:2229] T c465638422494c2ba970a42cfa14b76c P d57268eda08f4b679ade15c2c4b251c9 [term 1 LEADER]: Raft consensus shutting down.
I20260811 22:26:15.696753 13190 raft_consensus.cc:2258] T c465638422494c2ba970a42cfa14b76c P d57268eda08f4b679ade15c2c4b251c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:26:15.697906 13190 ts_tablet_manager.cc:1905] T c465638422494c2ba970a42cfa14b76c P d57268eda08f4b679ade15c2c4b251c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:26:15.704676 13190 ts_tablet_manager.cc:1918] T c465638422494c2ba970a42cfa14b76c P d57268eda08f4b679ade15c2c4b251c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 22:26:15.704818 13190 log.cc:1196] T c465638422494c2ba970a42cfa14b76c P d57268eda08f4b679ade15c2c4b251c9: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-1/wal/wals/c465638422494c2ba970a42cfa14b76c
I20260811 22:26:15.705171 13190 ts_tablet_manager.cc:1939] T c465638422494c2ba970a42cfa14b76c P d57268eda08f4b679ade15c2c4b251c9: Deleting consensus metadata
I20260811 22:26:15.706156 12204 catalog_manager.cc:4841] TS d57268eda08f4b679ade15c2c4b251c9 (127.29.106.2:39563): tablet c465638422494c2ba970a42cfa14b76c (table test-table [id=c8bb1172b0e9484c8025a55be36caafb]) successfully deleted
I20260811 22:26:15.706255 13192 ts_tablet_manager.cc:1918] T c465638422494c2ba970a42cfa14b76c P f3f6fc50d40445e3b7a35e0170a02880: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 22:26:15.706403 13192 log.cc:1196] T c465638422494c2ba970a42cfa14b76c P f3f6fc50d40445e3b7a35e0170a02880: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-0/wal/wals/c465638422494c2ba970a42cfa14b76c
I20260811 22:26:15.706815 13192 ts_tablet_manager.cc:1939] T c465638422494c2ba970a42cfa14b76c P f3f6fc50d40445e3b7a35e0170a02880: Deleting consensus metadata
I20260811 22:26:15.707473 13193 ts_tablet_manager.cc:1918] T c465638422494c2ba970a42cfa14b76c P 8f65ed2870d3440097c9b56016f85408: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 22:26:15.707634 13193 log.cc:1196] T c465638422494c2ba970a42cfa14b76c P 8f65ed2870d3440097c9b56016f85408: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-2/wal/wals/c465638422494c2ba970a42cfa14b76c
I20260811 22:26:15.707903 12207 catalog_manager.cc:4841] TS f3f6fc50d40445e3b7a35e0170a02880 (127.29.106.1:33759): tablet c465638422494c2ba970a42cfa14b76c (table test-table [id=c8bb1172b0e9484c8025a55be36caafb]) successfully deleted
I20260811 22:26:15.708042 13193 ts_tablet_manager.cc:1939] T c465638422494c2ba970a42cfa14b76c P 8f65ed2870d3440097c9b56016f85408: Deleting consensus metadata
I20260811 22:26:15.708911 12207 catalog_manager.cc:4841] TS 8f65ed2870d3440097c9b56016f85408 (127.29.106.3:38153): tablet c465638422494c2ba970a42cfa14b76c (table test-table [id=c8bb1172b0e9484c8025a55be36caafb]) successfully deleted
Aug 11 22:26:15 dist-test-slave-gjrz krb5kdc[12150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 11 22:26:15 dist-test-slave-gjrz krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:15 dist-test-slave-gjrz krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
W20260811 22:26:15.769666 12316 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:15.758582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41926 (local address 127.29.106.62:35049)
0811 22:26:15.758790 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:15.758797 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:15.759144 (+ 347us) server_negotiation.cc:407] Connection header received
0811 22:26:15.759331 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:15.759338 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:15.759425 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:15.759545 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:15.760757 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:15.761919 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:15.763142 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:15.763839 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:15.763967 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:15.769380 (+ 5413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41926: BlockingRecv error: recv got EOF from 127.0.0.1:41926 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 11 22:26:15 dist-test-slave-gjrz krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM
I20260811 22:26:15.786934 12316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:15.772446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41942 (local address 127.29.106.62:35049)
0811 22:26:15.772590 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:15.772594 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:15.773009 (+ 415us) server_negotiation.cc:407] Connection header received
0811 22:26:15.773129 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:15.773135 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:15.773222 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:15.773320 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:15.774397 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:15.775510 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:15.776736 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:15.777340 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:15.777464 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:15.781093 (+ 3629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:15.781100 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:15.781105 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:15.781145 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:15.784473 (+ 3328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:15.785065 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:15.785070 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:15.785074 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:15.785228 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:15.785544 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:15.785556 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:15.785559 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:15.785872 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:15.786097 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:15.786314 (+ 217us) server_negotiation.cc:299] Negotiation successful
0811 22:26:15.786568 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260811 22:26:15.790493 12225 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41942:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:26:15.791249 12225 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:26:15.798153 12470 tablet_service.cc:1467] Processing CreateTablet for tablet ab86365a53764520967c2bb67725e9f2 (DEFAULT_TABLE table=test-table [id=b8ca6847d6834c3a90886a2d893c8e0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:26:15.798604 12470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab86365a53764520967c2bb67725e9f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:26:15.799697 12731 tablet_service.cc:1467] Processing CreateTablet for tablet ab86365a53764520967c2bb67725e9f2 (DEFAULT_TABLE table=test-table [id=b8ca6847d6834c3a90886a2d893c8e0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:26:15.800168 12731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab86365a53764520967c2bb67725e9f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:26:15.803242 13149 tablet_bootstrap.cc:492] T ab86365a53764520967c2bb67725e9f2 P f3f6fc50d40445e3b7a35e0170a02880: Bootstrap starting.
I20260811 22:26:15.804620 13149 tablet_bootstrap.cc:654] T ab86365a53764520967c2bb67725e9f2 P f3f6fc50d40445e3b7a35e0170a02880: Neither blocks nor log segments found. Creating new log.
I20260811 22:26:15.806180 13149 tablet_bootstrap.cc:492] T ab86365a53764520967c2bb67725e9f2 P f3f6fc50d40445e3b7a35e0170a02880: No bootstrap required, opened a new log
I20260811 22:26:15.806314 13149 ts_tablet_manager.cc:1397] T ab86365a53764520967c2bb67725e9f2 P f3f6fc50d40445e3b7a35e0170a02880: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260811 22:26:15.807085 13151 tablet_bootstrap.cc:492] T ab86365a53764520967c2bb67725e9f2 P d57268eda08f4b679ade15c2c4b251c9: Bootstrap starting.
I20260811 22:26:15.806910 13149 raft_consensus.cc:348] T ab86365a53764520967c2bb67725e9f2 P f3f6fc50d40445e3b7a35e0170a02880 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } }
I20260811 22:26:15.807575 13149 raft_consensus.cc:374] T ab86365a53764520967c2bb67725e9f2 P f3f6fc50d40445e3b7a35e0170a02880 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:26:15.807725 13149 raft_consensus.cc:729] T ab86365a53764520967c2bb67725e9f2 P f3f6fc50d40445e3b7a35e0170a02880 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3f6fc50d40445e3b7a35e0170a02880, State: Initialized, Role: FOLLOWER
I20260811 22:26:15.807657 12968 tablet_service.cc:1467] Processing CreateTablet for tablet ab86365a53764520967c2bb67725e9f2 (DEFAULT_TABLE table=test-table [id=b8ca6847d6834c3a90886a2d893c8e0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:26:15.808084 12968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab86365a53764520967c2bb67725e9f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:26:15.807994 13149 consensus_queue.cc:260] T ab86365a53764520967c2bb67725e9f2 P f3f6fc50d40445e3b7a35e0170a02880 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } }
I20260811 22:26:15.808660 13149 ts_tablet_manager.cc:1428] T ab86365a53764520967c2bb67725e9f2 P f3f6fc50d40445e3b7a35e0170a02880: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260811 22:26:15.808816 13151 tablet_bootstrap.cc:654] T ab86365a53764520967c2bb67725e9f2 P d57268eda08f4b679ade15c2c4b251c9: Neither blocks nor log segments found. Creating new log.
I20260811 22:26:15.811326 13151 tablet_bootstrap.cc:492] T ab86365a53764520967c2bb67725e9f2 P d57268eda08f4b679ade15c2c4b251c9: No bootstrap required, opened a new log
I20260811 22:26:15.811471 13151 ts_tablet_manager.cc:1397] T ab86365a53764520967c2bb67725e9f2 P d57268eda08f4b679ade15c2c4b251c9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20260811 22:26:15.812618 13151 raft_consensus.cc:348] T ab86365a53764520967c2bb67725e9f2 P d57268eda08f4b679ade15c2c4b251c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } }
I20260811 22:26:15.812860 13151 raft_consensus.cc:374] T ab86365a53764520967c2bb67725e9f2 P d57268eda08f4b679ade15c2c4b251c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:26:15.812949 13151 raft_consensus.cc:729] T ab86365a53764520967c2bb67725e9f2 P d57268eda08f4b679ade15c2c4b251c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d57268eda08f4b679ade15c2c4b251c9, State: Initialized, Role: FOLLOWER
I20260811 22:26:15.813172 13151 consensus_queue.cc:260] T ab86365a53764520967c2bb67725e9f2 P d57268eda08f4b679ade15c2c4b251c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } }
I20260811 22:26:15.813694 13151 ts_tablet_manager.cc:1428] T ab86365a53764520967c2bb67725e9f2 P d57268eda08f4b679ade15c2c4b251c9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260811 22:26:15.815035 13152 tablet_bootstrap.cc:492] T ab86365a53764520967c2bb67725e9f2 P 8f65ed2870d3440097c9b56016f85408: Bootstrap starting.
I20260811 22:26:15.816601 13152 tablet_bootstrap.cc:654] T ab86365a53764520967c2bb67725e9f2 P 8f65ed2870d3440097c9b56016f85408: Neither blocks nor log segments found. Creating new log.
I20260811 22:26:15.818455 13152 tablet_bootstrap.cc:492] T ab86365a53764520967c2bb67725e9f2 P 8f65ed2870d3440097c9b56016f85408: No bootstrap required, opened a new log
I20260811 22:26:15.818612 13152 ts_tablet_manager.cc:1397] T ab86365a53764520967c2bb67725e9f2 P 8f65ed2870d3440097c9b56016f85408: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260811 22:26:15.819386 13152 raft_consensus.cc:348] T ab86365a53764520967c2bb67725e9f2 P 8f65ed2870d3440097c9b56016f85408 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } }
I20260811 22:26:15.819655 13152 raft_consensus.cc:374] T ab86365a53764520967c2bb67725e9f2 P 8f65ed2870d3440097c9b56016f85408 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:26:15.819748 13152 raft_consensus.cc:729] T ab86365a53764520967c2bb67725e9f2 P 8f65ed2870d3440097c9b56016f85408 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f65ed2870d3440097c9b56016f85408, State: Initialized, Role: FOLLOWER
I20260811 22:26:15.819989 13152 consensus_queue.cc:260] T ab86365a53764520967c2bb67725e9f2 P 8f65ed2870d3440097c9b56016f85408 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } }
I20260811 22:26:15.820508 13152 ts_tablet_manager.cc:1428] T ab86365a53764520967c2bb67725e9f2 P 8f65ed2870d3440097c9b56016f85408: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260811 22:26:15.936800 13247 raft_consensus.cc:482] T ab86365a53764520967c2bb67725e9f2 P d57268eda08f4b679ade15c2c4b251c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:26:15.937026 13247 raft_consensus.cc:504] T ab86365a53764520967c2bb67725e9f2 P d57268eda08f4b679ade15c2c4b251c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } }
I20260811 22:26:15.937775 13247 leader_election.cc:288] T ab86365a53764520967c2bb67725e9f2 P d57268eda08f4b679ade15c2c4b251c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3f6fc50d40445e3b7a35e0170a02880 (127.29.106.1:33759), 8f65ed2870d3440097c9b56016f85408 (127.29.106.3:38153)
I20260811 22:26:15.938364 12509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab86365a53764520967c2bb67725e9f2" candidate_uuid: "d57268eda08f4b679ade15c2c4b251c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3f6fc50d40445e3b7a35e0170a02880" is_pre_election: true
I20260811 22:26:15.938373 13015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab86365a53764520967c2bb67725e9f2" candidate_uuid: "d57268eda08f4b679ade15c2c4b251c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f65ed2870d3440097c9b56016f85408" is_pre_election: true
I20260811 22:26:15.938665 12509 raft_consensus.cc:2454] T ab86365a53764520967c2bb67725e9f2 P f3f6fc50d40445e3b7a35e0170a02880 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d57268eda08f4b679ade15c2c4b251c9 in term 0.
I20260811 22:26:15.938694 13015 raft_consensus.cc:2454] T ab86365a53764520967c2bb67725e9f2 P 8f65ed2870d3440097c9b56016f85408 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d57268eda08f4b679ade15c2c4b251c9 in term 0.
I20260811 22:26:15.939188 12650 leader_election.cc:302] T ab86365a53764520967c2bb67725e9f2 P d57268eda08f4b679ade15c2c4b251c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d57268eda08f4b679ade15c2c4b251c9, f3f6fc50d40445e3b7a35e0170a02880; no voters:
I20260811 22:26:15.939710 13247 raft_consensus.cc:2781] T ab86365a53764520967c2bb67725e9f2 P d57268eda08f4b679ade15c2c4b251c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:26:15.939838 13247 raft_consensus.cc:482] T ab86365a53764520967c2bb67725e9f2 P d57268eda08f4b679ade15c2c4b251c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:26:15.939939 13247 raft_consensus.cc:3037] T ab86365a53764520967c2bb67725e9f2 P d57268eda08f4b679ade15c2c4b251c9 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:26:15.943135 13247 raft_consensus.cc:504] T ab86365a53764520967c2bb67725e9f2 P d57268eda08f4b679ade15c2c4b251c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } }
I20260811 22:26:15.943807 13247 leader_election.cc:288] T ab86365a53764520967c2bb67725e9f2 P d57268eda08f4b679ade15c2c4b251c9 [CANDIDATE]: Term 1 election: Requested vote from peers f3f6fc50d40445e3b7a35e0170a02880 (127.29.106.1:33759), 8f65ed2870d3440097c9b56016f85408 (127.29.106.3:38153)
I20260811 22:26:15.944239 13015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab86365a53764520967c2bb67725e9f2" candidate_uuid: "d57268eda08f4b679ade15c2c4b251c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f65ed2870d3440097c9b56016f85408"
I20260811 22:26:15.944223 12509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab86365a53764520967c2bb67725e9f2" candidate_uuid: "d57268eda08f4b679ade15c2c4b251c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3f6fc50d40445e3b7a35e0170a02880"
I20260811 22:26:15.944425 13015 raft_consensus.cc:3037] T ab86365a53764520967c2bb67725e9f2 P 8f65ed2870d3440097c9b56016f85408 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:26:15.944433 12509 raft_consensus.cc:3037] T ab86365a53764520967c2bb67725e9f2 P f3f6fc50d40445e3b7a35e0170a02880 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:26:15.946399 13015 raft_consensus.cc:2454] T ab86365a53764520967c2bb67725e9f2 P 8f65ed2870d3440097c9b56016f85408 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d57268eda08f4b679ade15c2c4b251c9 in term 1.
I20260811 22:26:15.946777 12650 leader_election.cc:302] T ab86365a53764520967c2bb67725e9f2 P d57268eda08f4b679ade15c2c4b251c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f65ed2870d3440097c9b56016f85408, d57268eda08f4b679ade15c2c4b251c9; no voters:
I20260811 22:26:15.947031 13247 raft_consensus.cc:2781] T ab86365a53764520967c2bb67725e9f2 P d57268eda08f4b679ade15c2c4b251c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:26:15.947191 13247 raft_consensus.cc:686] T ab86365a53764520967c2bb67725e9f2 P d57268eda08f4b679ade15c2c4b251c9 [term 1 LEADER]: Becoming Leader. State: Replica: d57268eda08f4b679ade15c2c4b251c9, State: Running, Role: LEADER
I20260811 22:26:15.947358 12509 raft_consensus.cc:2454] T ab86365a53764520967c2bb67725e9f2 P f3f6fc50d40445e3b7a35e0170a02880 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d57268eda08f4b679ade15c2c4b251c9 in term 1.
I20260811 22:26:15.947495 13247 consensus_queue.cc:237] T ab86365a53764520967c2bb67725e9f2 P d57268eda08f4b679ade15c2c4b251c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } }
I20260811 22:26:15.950124 12228 catalog_manager.cc:5495] T ab86365a53764520967c2bb67725e9f2 P d57268eda08f4b679ade15c2c4b251c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to d57268eda08f4b679ade15c2c4b251c9 (127.29.106.2). New cstate: current_term: 1 leader_uuid: "d57268eda08f4b679ade15c2c4b251c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } health_report { overall_health: UNKNOWN } } }
I20260811 22:26:15.980409 13144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:15.975112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41958 (local address 127.29.106.2:39563)
0811 22:26:15.975286 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:15.975293 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:15.975312 (+ 19us) server_negotiation.cc:407] Connection header received
0811 22:26:15.975442 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:15.975445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:15.975526 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:15.975605 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:26:15.975997 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:15.976738 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:15.978661 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:15.979119 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:15.979205 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:15.979762 (+ 557us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:26:15.979878 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:26:15.979948 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:15.980080 (+ 132us) server_negotiation.cc:299] Negotiation successful
0811 22:26:15.980141 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260811 22:26:15.982925 12509 raft_consensus.cc:1261] T ab86365a53764520967c2bb67725e9f2 P f3f6fc50d40445e3b7a35e0170a02880 [term 1 FOLLOWER]: Refusing update from remote peer d57268eda08f4b679ade15c2c4b251c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:26:15.982980 13015 raft_consensus.cc:1261] T ab86365a53764520967c2bb67725e9f2 P 8f65ed2870d3440097c9b56016f85408 [term 1 FOLLOWER]: Refusing update from remote peer d57268eda08f4b679ade15c2c4b251c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:26:15.983448 13247 consensus_queue.cc:1035] T ab86365a53764520967c2bb67725e9f2 P d57268eda08f4b679ade15c2c4b251c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:26:15.983745 13158 consensus_queue.cc:1035] T ab86365a53764520967c2bb67725e9f2 P d57268eda08f4b679ade15c2c4b251c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:26:16.006959 12227 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:26:16.007226 12227 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:26:16.010519 12227 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0ecd4bd0e93d4f0f84a34aa8ecbe04fe: Sending DeleteTablet for 3 replicas of tablet ab86365a53764520967c2bb67725e9f2
I20260811 22:26:16.011255 12731 tablet_service.cc:1514] Processing DeleteTablet for tablet ab86365a53764520967c2bb67725e9f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:26:16 UTC) from {username='oryx', principal='oryx/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:41924
I20260811 22:26:16.011359 12470 tablet_service.cc:1514] Processing DeleteTablet for tablet ab86365a53764520967c2bb67725e9f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:26:16 UTC) from {username='oryx', principal='oryx/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:41724
I20260811 22:26:16.011584 13190 tablet_replica.cc:330] T ab86365a53764520967c2bb67725e9f2 P d57268eda08f4b679ade15c2c4b251c9: stopping tablet replica
I20260811 22:26:16.011617 12732 tablet_service.cc:1514] Processing DeleteTablet for tablet ab86365a53764520967c2bb67725e9f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:26:16 UTC) from {username='oryx', principal='oryx/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:41924
I20260811 22:26:16.011649 13192 tablet_replica.cc:330] T ab86365a53764520967c2bb67725e9f2 P f3f6fc50d40445e3b7a35e0170a02880: stopping tablet replica
I20260811 22:26:16.011773 13190 raft_consensus.cc:2229] T ab86365a53764520967c2bb67725e9f2 P d57268eda08f4b679ade15c2c4b251c9 [term 1 LEADER]: Raft consensus shutting down.
I20260811 22:26:16.011816 13192 raft_consensus.cc:2229] T ab86365a53764520967c2bb67725e9f2 P f3f6fc50d40445e3b7a35e0170a02880 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:26:16.012032 13192 raft_consensus.cc:2258] T ab86365a53764520967c2bb67725e9f2 P f3f6fc50d40445e3b7a35e0170a02880 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:26:16.012069 13190 raft_consensus.cc:2258] T ab86365a53764520967c2bb67725e9f2 P d57268eda08f4b679ade15c2c4b251c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:26:16.013157 13192 ts_tablet_manager.cc:1905] T ab86365a53764520967c2bb67725e9f2 P f3f6fc50d40445e3b7a35e0170a02880: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:26:16.013327 13190 ts_tablet_manager.cc:1905] T ab86365a53764520967c2bb67725e9f2 P d57268eda08f4b679ade15c2c4b251c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:26:16.017978 13190 ts_tablet_manager.cc:1918] T ab86365a53764520967c2bb67725e9f2 P d57268eda08f4b679ade15c2c4b251c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:26:16.018110 13190 log.cc:1196] T ab86365a53764520967c2bb67725e9f2 P d57268eda08f4b679ade15c2c4b251c9: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-1/wal/wals/ab86365a53764520967c2bb67725e9f2
I20260811 22:26:16.018525 13190 ts_tablet_manager.cc:1939] T ab86365a53764520967c2bb67725e9f2 P d57268eda08f4b679ade15c2c4b251c9: Deleting consensus metadata
I20260811 22:26:16.018899 12968 tablet_service.cc:1514] Processing DeleteTablet for tablet ab86365a53764520967c2bb67725e9f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:26:16 UTC) from {username='oryx', principal='oryx/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:50906
I20260811 22:26:16.019181 13193 tablet_replica.cc:330] T ab86365a53764520967c2bb67725e9f2 P 8f65ed2870d3440097c9b56016f85408: stopping tablet replica
I20260811 22:26:16.019377 13193 raft_consensus.cc:2229] T ab86365a53764520967c2bb67725e9f2 P 8f65ed2870d3440097c9b56016f85408 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:26:16.019634 13192 ts_tablet_manager.cc:1918] T ab86365a53764520967c2bb67725e9f2 P f3f6fc50d40445e3b7a35e0170a02880: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:26:16.019645 13193 raft_consensus.cc:2258] T ab86365a53764520967c2bb67725e9f2 P 8f65ed2870d3440097c9b56016f85408 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:26:16.019622 12204 catalog_manager.cc:4841] TS d57268eda08f4b679ade15c2c4b251c9 (127.29.106.2:39563): tablet ab86365a53764520967c2bb67725e9f2 (table test-table [id=b8ca6847d6834c3a90886a2d893c8e0b]) successfully deleted
I20260811 22:26:16.019753 13192 log.cc:1196] T ab86365a53764520967c2bb67725e9f2 P f3f6fc50d40445e3b7a35e0170a02880: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-0/wal/wals/ab86365a53764520967c2bb67725e9f2
I20260811 22:26:16.020166 13192 ts_tablet_manager.cc:1939] T ab86365a53764520967c2bb67725e9f2 P f3f6fc50d40445e3b7a35e0170a02880: Deleting consensus metadata
W20260811 22:26:16.020268 12204 catalog_manager.cc:4808] TS d57268eda08f4b679ade15c2c4b251c9 (127.29.106.2:39563): delete failed for tablet ab86365a53764520967c2bb67725e9f2 because the tablet was not found. No further retry: Not found: Tablet not found: ab86365a53764520967c2bb67725e9f2
I20260811 22:26:16.021200 13193 ts_tablet_manager.cc:1905] T ab86365a53764520967c2bb67725e9f2 P 8f65ed2870d3440097c9b56016f85408: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:26:16.021811 12207 catalog_manager.cc:4841] TS f3f6fc50d40445e3b7a35e0170a02880 (127.29.106.1:33759): tablet ab86365a53764520967c2bb67725e9f2 (table test-table [id=b8ca6847d6834c3a90886a2d893c8e0b]) successfully deleted
I20260811 22:26:16.022620 12227 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41942:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:26:16.023308 12227 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:26:16.026829 13193 ts_tablet_manager.cc:1918] T ab86365a53764520967c2bb67725e9f2 P 8f65ed2870d3440097c9b56016f85408: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:26:16.026942 13193 log.cc:1196] T ab86365a53764520967c2bb67725e9f2 P 8f65ed2870d3440097c9b56016f85408: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-2/wal/wals/ab86365a53764520967c2bb67725e9f2
I20260811 22:26:16.027288 13193 ts_tablet_manager.cc:1939] T ab86365a53764520967c2bb67725e9f2 P 8f65ed2870d3440097c9b56016f85408: Deleting consensus metadata
I20260811 22:26:16.028296 12207 catalog_manager.cc:4841] TS 8f65ed2870d3440097c9b56016f85408 (127.29.106.3:38153): tablet ab86365a53764520967c2bb67725e9f2 (table test-table [id=b8ca6847d6834c3a90886a2d893c8e0b]) successfully deleted
I20260811 22:26:16.030180 12470 tablet_service.cc:1467] Processing CreateTablet for tablet 6156f50419fd41fdaa05d8e0a52b9268 (DEFAULT_TABLE table=test-table [id=1127aed1eb6e4a0fbdb13fcb156fb32d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:26:16.030340 12968 tablet_service.cc:1467] Processing CreateTablet for tablet 6156f50419fd41fdaa05d8e0a52b9268 (DEFAULT_TABLE table=test-table [id=1127aed1eb6e4a0fbdb13fcb156fb32d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:26:16.030625 12470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6156f50419fd41fdaa05d8e0a52b9268. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:26:16.030755 12968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6156f50419fd41fdaa05d8e0a52b9268. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:26:16.035408 13152 tablet_bootstrap.cc:492] T 6156f50419fd41fdaa05d8e0a52b9268 P 8f65ed2870d3440097c9b56016f85408: Bootstrap starting.
I20260811 22:26:16.035732 12731 tablet_service.cc:1467] Processing CreateTablet for tablet 6156f50419fd41fdaa05d8e0a52b9268 (DEFAULT_TABLE table=test-table [id=1127aed1eb6e4a0fbdb13fcb156fb32d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:26:16.036177 12731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6156f50419fd41fdaa05d8e0a52b9268. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:26:16.036897 13149 tablet_bootstrap.cc:492] T 6156f50419fd41fdaa05d8e0a52b9268 P f3f6fc50d40445e3b7a35e0170a02880: Bootstrap starting.
I20260811 22:26:16.036978 13152 tablet_bootstrap.cc:654] T 6156f50419fd41fdaa05d8e0a52b9268 P 8f65ed2870d3440097c9b56016f85408: Neither blocks nor log segments found. Creating new log.
I20260811 22:26:16.038384 13149 tablet_bootstrap.cc:654] T 6156f50419fd41fdaa05d8e0a52b9268 P f3f6fc50d40445e3b7a35e0170a02880: Neither blocks nor log segments found. Creating new log.
I20260811 22:26:16.044814 13151 tablet_bootstrap.cc:492] T 6156f50419fd41fdaa05d8e0a52b9268 P d57268eda08f4b679ade15c2c4b251c9: Bootstrap starting.
I20260811 22:26:16.046124 13151 tablet_bootstrap.cc:654] T 6156f50419fd41fdaa05d8e0a52b9268 P d57268eda08f4b679ade15c2c4b251c9: Neither blocks nor log segments found. Creating new log.
I20260811 22:26:16.050300 13149 tablet_bootstrap.cc:492] T 6156f50419fd41fdaa05d8e0a52b9268 P f3f6fc50d40445e3b7a35e0170a02880: No bootstrap required, opened a new log
I20260811 22:26:16.050480 13149 ts_tablet_manager.cc:1397] T 6156f50419fd41fdaa05d8e0a52b9268 P f3f6fc50d40445e3b7a35e0170a02880: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260811 22:26:16.050742 13151 tablet_bootstrap.cc:492] T 6156f50419fd41fdaa05d8e0a52b9268 P d57268eda08f4b679ade15c2c4b251c9: No bootstrap required, opened a new log
I20260811 22:26:16.050909 13151 ts_tablet_manager.cc:1397] T 6156f50419fd41fdaa05d8e0a52b9268 P d57268eda08f4b679ade15c2c4b251c9: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20260811 22:26:16.051188 13149 raft_consensus.cc:348] T 6156f50419fd41fdaa05d8e0a52b9268 P f3f6fc50d40445e3b7a35e0170a02880 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } }
I20260811 22:26:16.051463 13149 raft_consensus.cc:374] T 6156f50419fd41fdaa05d8e0a52b9268 P f3f6fc50d40445e3b7a35e0170a02880 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:26:16.051548 13149 raft_consensus.cc:729] T 6156f50419fd41fdaa05d8e0a52b9268 P f3f6fc50d40445e3b7a35e0170a02880 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3f6fc50d40445e3b7a35e0170a02880, State: Initialized, Role: FOLLOWER
I20260811 22:26:16.051800 13152 tablet_bootstrap.cc:492] T 6156f50419fd41fdaa05d8e0a52b9268 P 8f65ed2870d3440097c9b56016f85408: No bootstrap required, opened a new log
I20260811 22:26:16.051772 13149 consensus_queue.cc:260] T 6156f50419fd41fdaa05d8e0a52b9268 P f3f6fc50d40445e3b7a35e0170a02880 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } }
I20260811 22:26:16.051749 13151 raft_consensus.cc:348] T 6156f50419fd41fdaa05d8e0a52b9268 P d57268eda08f4b679ade15c2c4b251c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } }
I20260811 22:26:16.052023 13151 raft_consensus.cc:374] T 6156f50419fd41fdaa05d8e0a52b9268 P d57268eda08f4b679ade15c2c4b251c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:26:16.052112 13151 raft_consensus.cc:729] T 6156f50419fd41fdaa05d8e0a52b9268 P d57268eda08f4b679ade15c2c4b251c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d57268eda08f4b679ade15c2c4b251c9, State: Initialized, Role: FOLLOWER
I20260811 22:26:16.052127 13152 ts_tablet_manager.cc:1397] T 6156f50419fd41fdaa05d8e0a52b9268 P 8f65ed2870d3440097c9b56016f85408: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.001s
I20260811 22:26:16.052348 13151 consensus_queue.cc:260] T 6156f50419fd41fdaa05d8e0a52b9268 P d57268eda08f4b679ade15c2c4b251c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } }
I20260811 22:26:16.052891 13149 ts_tablet_manager.cc:1428] T 6156f50419fd41fdaa05d8e0a52b9268 P f3f6fc50d40445e3b7a35e0170a02880: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260811 22:26:16.053426 13152 raft_consensus.cc:348] T 6156f50419fd41fdaa05d8e0a52b9268 P 8f65ed2870d3440097c9b56016f85408 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } }
I20260811 22:26:16.053723 13152 raft_consensus.cc:374] T 6156f50419fd41fdaa05d8e0a52b9268 P 8f65ed2870d3440097c9b56016f85408 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:26:16.053853 13152 raft_consensus.cc:729] T 6156f50419fd41fdaa05d8e0a52b9268 P 8f65ed2870d3440097c9b56016f85408 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f65ed2870d3440097c9b56016f85408, State: Initialized, Role: FOLLOWER
I20260811 22:26:16.054178 13152 consensus_queue.cc:260] T 6156f50419fd41fdaa05d8e0a52b9268 P 8f65ed2870d3440097c9b56016f85408 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } }
I20260811 22:26:16.054836 13151 ts_tablet_manager.cc:1428] T 6156f50419fd41fdaa05d8e0a52b9268 P d57268eda08f4b679ade15c2c4b251c9: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260811 22:26:16.057745 13152 ts_tablet_manager.cc:1428] T 6156f50419fd41fdaa05d8e0a52b9268 P 8f65ed2870d3440097c9b56016f85408: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260811 22:26:16.196503 13247 raft_consensus.cc:482] T 6156f50419fd41fdaa05d8e0a52b9268 P d57268eda08f4b679ade15c2c4b251c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:26:16.196725 13247 raft_consensus.cc:504] T 6156f50419fd41fdaa05d8e0a52b9268 P d57268eda08f4b679ade15c2c4b251c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } }
I20260811 22:26:16.197444 13247 leader_election.cc:288] T 6156f50419fd41fdaa05d8e0a52b9268 P d57268eda08f4b679ade15c2c4b251c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3f6fc50d40445e3b7a35e0170a02880 (127.29.106.1:33759), 8f65ed2870d3440097c9b56016f85408 (127.29.106.3:38153)
I20260811 22:26:16.198060 12509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6156f50419fd41fdaa05d8e0a52b9268" candidate_uuid: "d57268eda08f4b679ade15c2c4b251c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3f6fc50d40445e3b7a35e0170a02880" is_pre_election: true
I20260811 22:26:16.198104 13015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6156f50419fd41fdaa05d8e0a52b9268" candidate_uuid: "d57268eda08f4b679ade15c2c4b251c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f65ed2870d3440097c9b56016f85408" is_pre_election: true
I20260811 22:26:16.198364 13015 raft_consensus.cc:2454] T 6156f50419fd41fdaa05d8e0a52b9268 P 8f65ed2870d3440097c9b56016f85408 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d57268eda08f4b679ade15c2c4b251c9 in term 0.
I20260811 22:26:16.198365 12509 raft_consensus.cc:2454] T 6156f50419fd41fdaa05d8e0a52b9268 P f3f6fc50d40445e3b7a35e0170a02880 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d57268eda08f4b679ade15c2c4b251c9 in term 0.
I20260811 22:26:16.198841 12650 leader_election.cc:302] T 6156f50419fd41fdaa05d8e0a52b9268 P d57268eda08f4b679ade15c2c4b251c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f65ed2870d3440097c9b56016f85408, d57268eda08f4b679ade15c2c4b251c9; no voters:
I20260811 22:26:16.199163 13247 raft_consensus.cc:2781] T 6156f50419fd41fdaa05d8e0a52b9268 P d57268eda08f4b679ade15c2c4b251c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:26:16.199333 13247 raft_consensus.cc:482] T 6156f50419fd41fdaa05d8e0a52b9268 P d57268eda08f4b679ade15c2c4b251c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:26:16.199448 13247 raft_consensus.cc:3037] T 6156f50419fd41fdaa05d8e0a52b9268 P d57268eda08f4b679ade15c2c4b251c9 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:26:16.202610 13247 raft_consensus.cc:504] T 6156f50419fd41fdaa05d8e0a52b9268 P d57268eda08f4b679ade15c2c4b251c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } }
I20260811 22:26:16.203306 13247 leader_election.cc:288] T 6156f50419fd41fdaa05d8e0a52b9268 P d57268eda08f4b679ade15c2c4b251c9 [CANDIDATE]: Term 1 election: Requested vote from peers f3f6fc50d40445e3b7a35e0170a02880 (127.29.106.1:33759), 8f65ed2870d3440097c9b56016f85408 (127.29.106.3:38153)
I20260811 22:26:16.203744 12509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6156f50419fd41fdaa05d8e0a52b9268" candidate_uuid: "d57268eda08f4b679ade15c2c4b251c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3f6fc50d40445e3b7a35e0170a02880"
I20260811 22:26:16.203858 13015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6156f50419fd41fdaa05d8e0a52b9268" candidate_uuid: "d57268eda08f4b679ade15c2c4b251c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f65ed2870d3440097c9b56016f85408"
I20260811 22:26:16.203977 12509 raft_consensus.cc:3037] T 6156f50419fd41fdaa05d8e0a52b9268 P f3f6fc50d40445e3b7a35e0170a02880 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:26:16.204121 13015 raft_consensus.cc:3037] T 6156f50419fd41fdaa05d8e0a52b9268 P 8f65ed2870d3440097c9b56016f85408 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:26:16.206866 12509 raft_consensus.cc:2454] T 6156f50419fd41fdaa05d8e0a52b9268 P f3f6fc50d40445e3b7a35e0170a02880 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d57268eda08f4b679ade15c2c4b251c9 in term 1.
I20260811 22:26:16.206866 13015 raft_consensus.cc:2454] T 6156f50419fd41fdaa05d8e0a52b9268 P 8f65ed2870d3440097c9b56016f85408 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d57268eda08f4b679ade15c2c4b251c9 in term 1.
I20260811 22:26:16.207254 12650 leader_election.cc:302] T 6156f50419fd41fdaa05d8e0a52b9268 P d57268eda08f4b679ade15c2c4b251c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f65ed2870d3440097c9b56016f85408, d57268eda08f4b679ade15c2c4b251c9; no voters:
I20260811 22:26:16.207597 13247 raft_consensus.cc:2781] T 6156f50419fd41fdaa05d8e0a52b9268 P d57268eda08f4b679ade15c2c4b251c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:26:16.207751 13247 raft_consensus.cc:686] T 6156f50419fd41fdaa05d8e0a52b9268 P d57268eda08f4b679ade15c2c4b251c9 [term 1 LEADER]: Becoming Leader. State: Replica: d57268eda08f4b679ade15c2c4b251c9, State: Running, Role: LEADER
I20260811 22:26:16.208022 13247 consensus_queue.cc:237] T 6156f50419fd41fdaa05d8e0a52b9268 P d57268eda08f4b679ade15c2c4b251c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } }
I20260811 22:26:16.210884 12228 catalog_manager.cc:5495] T 6156f50419fd41fdaa05d8e0a52b9268 P d57268eda08f4b679ade15c2c4b251c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to d57268eda08f4b679ade15c2c4b251c9 (127.29.106.2). New cstate: current_term: 1 leader_uuid: "d57268eda08f4b679ade15c2c4b251c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d57268eda08f4b679ade15c2c4b251c9" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 } health_report { overall_health: UNKNOWN } } }
I20260811 22:26:16.255712 12507 raft_consensus.cc:1261] T 6156f50419fd41fdaa05d8e0a52b9268 P f3f6fc50d40445e3b7a35e0170a02880 [term 1 FOLLOWER]: Refusing update from remote peer d57268eda08f4b679ade15c2c4b251c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:26:16.255916 13015 raft_consensus.cc:1261] T 6156f50419fd41fdaa05d8e0a52b9268 P 8f65ed2870d3440097c9b56016f85408 [term 1 FOLLOWER]: Refusing update from remote peer d57268eda08f4b679ade15c2c4b251c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:26:16.256578 13200 consensus_queue.cc:1035] T 6156f50419fd41fdaa05d8e0a52b9268 P d57268eda08f4b679ade15c2c4b251c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f65ed2870d3440097c9b56016f85408" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 38153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:26:16.256928 13247 consensus_queue.cc:1035] T 6156f50419fd41fdaa05d8e0a52b9268 P d57268eda08f4b679ade15c2c4b251c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3f6fc50d40445e3b7a35e0170a02880" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 33759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:26:16.308779 12228 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:26:16.309060 12228 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:26:16.312781 12228 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0ecd4bd0e93d4f0f84a34aa8ecbe04fe: Sending DeleteTablet for 3 replicas of tablet 6156f50419fd41fdaa05d8e0a52b9268
I20260811 22:26:16.313580 12731 tablet_service.cc:1514] Processing DeleteTablet for tablet 6156f50419fd41fdaa05d8e0a52b9268 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:26:16 UTC) from {username='oryx', principal='oryx/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:41924
I20260811 22:26:16.314080 12470 tablet_service.cc:1514] Processing DeleteTablet for tablet 6156f50419fd41fdaa05d8e0a52b9268 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:26:16 UTC) from {username='oryx', principal='oryx/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:41724
I20260811 22:26:16.314136 13190 tablet_replica.cc:330] T 6156f50419fd41fdaa05d8e0a52b9268 P d57268eda08f4b679ade15c2c4b251c9: stopping tablet replica
I20260811 22:26:16.314370 13192 tablet_replica.cc:330] T 6156f50419fd41fdaa05d8e0a52b9268 P f3f6fc50d40445e3b7a35e0170a02880: stopping tablet replica
I20260811 22:26:16.314366 13190 raft_consensus.cc:2229] T 6156f50419fd41fdaa05d8e0a52b9268 P d57268eda08f4b679ade15c2c4b251c9 [term 1 LEADER]: Raft consensus shutting down.
I20260811 22:26:16.314587 13192 raft_consensus.cc:2229] T 6156f50419fd41fdaa05d8e0a52b9268 P f3f6fc50d40445e3b7a35e0170a02880 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:26:16.314724 13190 raft_consensus.cc:2258] T 6156f50419fd41fdaa05d8e0a52b9268 P d57268eda08f4b679ade15c2c4b251c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:26:16.314792 13192 raft_consensus.cc:2258] T 6156f50419fd41fdaa05d8e0a52b9268 P f3f6fc50d40445e3b7a35e0170a02880 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:26:16.315037 12968 tablet_service.cc:1514] Processing DeleteTablet for tablet 6156f50419fd41fdaa05d8e0a52b9268 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:26:16 UTC) from {username='oryx', principal='oryx/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:50906
I20260811 22:26:16.319175 13193 tablet_replica.cc:330] T 6156f50419fd41fdaa05d8e0a52b9268 P 8f65ed2870d3440097c9b56016f85408: stopping tablet replica
I20260811 22:26:16.319393 13193 raft_consensus.cc:2229] T 6156f50419fd41fdaa05d8e0a52b9268 P 8f65ed2870d3440097c9b56016f85408 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:26:16.319579 13193 raft_consensus.cc:2258] T 6156f50419fd41fdaa05d8e0a52b9268 P 8f65ed2870d3440097c9b56016f85408 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:26:16.319705 13192 ts_tablet_manager.cc:1905] T 6156f50419fd41fdaa05d8e0a52b9268 P f3f6fc50d40445e3b7a35e0170a02880: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:26:16.322335 13193 ts_tablet_manager.cc:1905] T 6156f50419fd41fdaa05d8e0a52b9268 P 8f65ed2870d3440097c9b56016f85408: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:26:16.322393 13190 ts_tablet_manager.cc:1905] T 6156f50419fd41fdaa05d8e0a52b9268 P d57268eda08f4b679ade15c2c4b251c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:26:16.327886 13192 ts_tablet_manager.cc:1918] T 6156f50419fd41fdaa05d8e0a52b9268 P f3f6fc50d40445e3b7a35e0170a02880: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 22:26:16.328044 13192 log.cc:1196] T 6156f50419fd41fdaa05d8e0a52b9268 P f3f6fc50d40445e3b7a35e0170a02880: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-0/wal/wals/6156f50419fd41fdaa05d8e0a52b9268
I20260811 22:26:16.328498 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 12357
I20260811 22:26:16.328725 13192 ts_tablet_manager.cc:1939] T 6156f50419fd41fdaa05d8e0a52b9268 P f3f6fc50d40445e3b7a35e0170a02880: Deleting consensus metadata
I20260811 22:26:16.329401 13193 ts_tablet_manager.cc:1918] T 6156f50419fd41fdaa05d8e0a52b9268 P 8f65ed2870d3440097c9b56016f85408: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 22:26:16.329595 13193 log.cc:1196] T 6156f50419fd41fdaa05d8e0a52b9268 P 8f65ed2870d3440097c9b56016f85408: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-2/wal/wals/6156f50419fd41fdaa05d8e0a52b9268
I20260811 22:26:16.330014 13193 ts_tablet_manager.cc:1939] T 6156f50419fd41fdaa05d8e0a52b9268 P 8f65ed2870d3440097c9b56016f85408: Deleting consensus metadata
I20260811 22:26:16.331907 12207 catalog_manager.cc:4841] TS 8f65ed2870d3440097c9b56016f85408 (127.29.106.3:38153): tablet 6156f50419fd41fdaa05d8e0a52b9268 (table test-table [id=1127aed1eb6e4a0fbdb13fcb156fb32d]) successfully deleted
I20260811 22:26:16.333503 12207 catalog_manager.cc:4841] TS f3f6fc50d40445e3b7a35e0170a02880 (127.29.106.1:33759): tablet 6156f50419fd41fdaa05d8e0a52b9268 (table test-table [id=1127aed1eb6e4a0fbdb13fcb156fb32d]) successfully deleted
I20260811 22:26:16.334949 13190 ts_tablet_manager.cc:1918] T 6156f50419fd41fdaa05d8e0a52b9268 P d57268eda08f4b679ade15c2c4b251c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 22:26:16.335175 13190 log.cc:1196] T 6156f50419fd41fdaa05d8e0a52b9268 P d57268eda08f4b679ade15c2c4b251c9: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786487105757681-30120-0/minicluster-data/ts-1/wal/wals/6156f50419fd41fdaa05d8e0a52b9268
I20260811 22:26:16.335873 13190 ts_tablet_manager.cc:1939] T 6156f50419fd41fdaa05d8e0a52b9268 P d57268eda08f4b679ade15c2c4b251c9: Deleting consensus metadata
I20260811 22:26:16.336833 12204 catalog_manager.cc:4841] TS d57268eda08f4b679ade15c2c4b251c9 (127.29.106.2:39563): tablet 6156f50419fd41fdaa05d8e0a52b9268 (table test-table [id=1127aed1eb6e4a0fbdb13fcb156fb32d]) successfully deleted
Aug 11 22:26:16 dist-test-slave-gjrz krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.106.1@KRBTEST.COM
I20260811 22:26:16.354719 13313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:16.332353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41732 (local address 127.29.106.1:33759)
0811 22:26:16.339148 (+ 6795us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:16.339153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:16.339168 (+ 15us) server_negotiation.cc:407] Connection header received
0811 22:26:16.339215 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:16.339218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:16.339353 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:16.339451 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:16.341158 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:16.341959 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:16.343810 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:16.344223 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:16.344321 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:16.349231 (+ 4910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:16.349246 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:16.349265 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:16.349295 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:16.352062 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:16.352806 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:16.352812 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:16.352815 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:16.352965 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:16.353322 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:16.353325 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:16.353327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:16.353671 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:16.353941 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:16.354283 (+ 342us) server_negotiation.cc:299] Negotiation successful
0811 22:26:16.354450 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6679,"thread_start_us":6568,"threads_started":1}
I20260811 22:26:16.355537 12595 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:26:16.478458 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 12357
I20260811 22:26:16.504292 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 12609
Aug 11 22:26:16 dist-test-slave-gjrz krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.106.2@KRBTEST.COM
I20260811 22:26:16.518982 13319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:16.505025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41966 (local address 127.29.106.2:39563)
0811 22:26:16.505511 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:16.505516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:16.505531 (+ 15us) server_negotiation.cc:407] Connection header received
0811 22:26:16.505595 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:16.505599 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:16.505666 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:16.505758 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:16.506991 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:16.507889 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:16.509190 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:16.509607 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:16.509717 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:16.513738 (+ 4021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:16.513752 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:16.513768 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:16.513800 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:16.516349 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:16.516961 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:16.516969 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:16.516971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:16.517104 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:16.517452 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:16.517455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:16.517458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:16.517804 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:16.518103 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:16.518424 (+ 321us) server_negotiation.cc:299] Negotiation successful
0811 22:26:16.518660 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":250,"threads_started":1}
I20260811 22:26:16.519716 12850 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:26:16.666126 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 12609
I20260811 22:26:16.696141 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 12863
Aug 11 22:26:16 dist-test-slave-gjrz krb5kdc[12150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.106.3@KRBTEST.COM
I20260811 22:26:16.711153 13323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:16.696928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50934 (local address 127.29.106.3:38153)
0811 22:26:16.697405 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:16.697409 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:16.697425 (+ 16us) server_negotiation.cc:407] Connection header received
0811 22:26:16.697479 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:16.697483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:16.697542 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:16.697637 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:16.698972 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:16.699755 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:16.701031 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:16.701444 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:16.701555 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:16.705824 (+ 4269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:16.705839 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:16.705857 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:16.705903 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:16.708698 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:16.709330 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:16.709338 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:16.709340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:16.709484 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:16.709787 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:16.709791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:16.709793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:16.710176 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:16.710403 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:16.710607 (+ 204us) server_negotiation.cc:299] Negotiation successful
0811 22:26:16.710778 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":249,"threads_started":1}
I20260811 22:26:16.711807 13092 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:26:16.833329 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 12863
I20260811 22:26:16.864953 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 12170
I20260811 22:26:16.866147 12302 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:26:16.983747 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 12170
2026-08-11T22:26:17Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6635 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786487105757681-30120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:26:17 dist-test-slave-gjrz krb5kdc[13328](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:26:17 dist-test-slave-gjrz krb5kdc[13328](info): set up 2 sockets
Aug 11 22:26:17 dist-test-slave-gjrz krb5kdc[13328](info): commencing operation
krb5kdc: starting...
W20260811 22:26:18.243953 30120 mini_kdc.cc:121] Time spent starting KDC: real 1.207s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Aug 11 22:26:18 dist-test-slave-gjrz krb5kdc[13328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:26:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:26:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.106.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
I20260811 22:26:18.471019 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:33653
--webserver_interface=127.29.106.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:41827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:33653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:26:18.793658 13353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:26:18.793915 13353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:26:18.794013 13353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:26:18.800984 13353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:26:18.801082 13353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:26:18.801136 13353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:26:18.801190 13353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:26:18.801239 13353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:26:18.810622 13353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:41827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:33653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:33653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.106.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:26:18.811791 13353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:26:18.812987 13353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:26:18.824162 13365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:18.824338 13363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:18.824699 13361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:26:18.824959 13353 server_base.cc:1034] running on GCE node
I20260811 22:26:18.825649 13353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:26:18.826916 13353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:26:18.828118 13353 hybrid_clock.cc:648] HybridClock initialized: now 1786487178828062 us; error 44 us; skew 500 ppm
I20260811 22:26:18.828429 13353 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-11T22:26:18Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1912 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:26:19 dist-test-slave-gjrz krb5kdc[13378](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:26:19 dist-test-slave-gjrz krb5kdc[13378](info): set up 2 sockets
Aug 11 22:26:19 dist-test-slave-gjrz krb5kdc[13378](info): commencing operation
krb5kdc: starting...
W20260811 22:26:20.151852 30120 mini_kdc.cc:121] Time spent starting KDC: real 1.202s user 0.006s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Aug 11 22:26:20 dist-test-slave-gjrz krb5kdc[13378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:26:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:26:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.106.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
I20260811 22:26:20.378547 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:42979
--webserver_interface=127.29.106.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:34383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:42979
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:26:20.698760 13403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:26:20.699007 13403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:26:20.699119 13403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:26:20.706007 13403 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:26:20.706099 13403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:26:20.706168 13403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:26:20.706240 13403 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:26:20.706297 13403 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:26:20.715633 13403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:34383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:42979
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:42979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.106.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:26:20.716773 13403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:26:20.717959 13403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:26:20.729094 13412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:20.729064 13416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:20.729111 13413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:26:20.730124 13403 server_base.cc:1034] running on GCE node
I20260811 22:26:20.730845 13403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:26:20.732187 13403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:26:20.733379 13403 hybrid_clock.cc:648] HybridClock initialized: now 1786487180733338 us; error 30 us; skew 500 ppm
I20260811 22:26:20.733696 13403 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:26:20 dist-test-slave-gjrz krb5kdc[13378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487180, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:26:20.736615 13403 init.cc:377] Logged in from keytab as kudu/127.29.106.62@KRBTEST.COM (short username kudu)
I20260811 22:26:20.738072 13403 webserver.cc:460] Webserver started at http://127.29.106.62:36249/ using document root <none> and password file <none>
I20260811 22:26:20.738843 13403 fs_manager.cc:359] Metadata directory not provided
I20260811 22:26:20.738970 13403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:26:20.739291 13403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:26:20.742057 13403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "0c4a02e8a46a4a30a815a25d21e966ff"
format_stamp: "Formatted at 2026-08-11 22:26:20 on dist-test-slave-gjrz"
server_key: "1159e82266cfda43355ae2fa9362a535"
server_key_iv: "3a3a244bbd1add8c9aad96758e6103cb"
server_key_version: "encryptionkey@0"
I20260811 22:26:20.742779 13403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "0c4a02e8a46a4a30a815a25d21e966ff"
format_stamp: "Formatted at 2026-08-11 22:26:20 on dist-test-slave-gjrz"
server_key: "1159e82266cfda43355ae2fa9362a535"
server_key_iv: "3a3a244bbd1add8c9aad96758e6103cb"
server_key_version: "encryptionkey@0"
I20260811 22:26:20.747623 13403 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260811 22:26:20.750837 13430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:20.752079 13403 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:26:20.752233 13403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "0c4a02e8a46a4a30a815a25d21e966ff"
format_stamp: "Formatted at 2026-08-11 22:26:20 on dist-test-slave-gjrz"
server_key: "1159e82266cfda43355ae2fa9362a535"
server_key_iv: "3a3a244bbd1add8c9aad96758e6103cb"
server_key_version: "encryptionkey@0"
I20260811 22:26:20.752398 13403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:26:20.769055 13403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:26:20.772657 13403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:26:20.772890 13403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:26:20.790061 13403 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.62:42979
I20260811 22:26:20.790086 13530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.62:42979 every 8 connection(s)
I20260811 22:26:20.791250 13403 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:26:20.794728 13531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:26:20.798794 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 13403
I20260811 22:26:20.799031 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:26:20.799824 30120 external_mini_cluster.cc:1428] Setting key 3b73c2084ce5f0691f70c8d0b9488f1f
I20260811 22:26:20.805778 13531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c4a02e8a46a4a30a815a25d21e966ff: Bootstrap starting.
I20260811 22:26:20.809238 13531 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c4a02e8a46a4a30a815a25d21e966ff: Neither blocks nor log segments found. Creating new log.
I20260811 22:26:20.810382 13531 log.cc:824] T 00000000000000000000000000000000 P 0c4a02e8a46a4a30a815a25d21e966ff: Log is configured to *not* fsync() on all Append() calls
I20260811 22:26:20.812949 13531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c4a02e8a46a4a30a815a25d21e966ff: No bootstrap required, opened a new log
Aug 11 22:26:20 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:26:20.821568 13535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:20.801667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54326 (local address 127.29.106.62:42979)
0811 22:26:20.802401 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:20.802411 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:20.802459 (+ 48us) server_negotiation.cc:407] Connection header received
0811 22:26:20.804444 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:20.804465 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:20.804795 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:20.805105 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:20.806923 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:20.808267 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:20.810048 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:20.810736 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:20.810923 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:20.814706 (+ 3783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:20.814714 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:20.814727 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:20.814763 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:20.817747 (+ 2984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:20.818517 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:20.818523 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:20.818529 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:20.818696 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:20.819067 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:20.819070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:20.819072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:20.819641 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:20.819932 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:20.820291 (+ 359us) server_negotiation.cc:299] Negotiation successful
0811 22:26:20.820615 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":357,"threads_started":1}
I20260811 22:26:20.821630 13531 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c4a02e8a46a4a30a815a25d21e966ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c4a02e8a46a4a30a815a25d21e966ff" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 42979 } }
I20260811 22:26:20.822013 13531 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c4a02e8a46a4a30a815a25d21e966ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:26:20.822122 13531 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c4a02e8a46a4a30a815a25d21e966ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c4a02e8a46a4a30a815a25d21e966ff, State: Initialized, Role: FOLLOWER
I20260811 22:26:20.822674 13531 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c4a02e8a46a4a30a815a25d21e966ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c4a02e8a46a4a30a815a25d21e966ff" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 42979 } }
I20260811 22:26:20.822948 13531 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c4a02e8a46a4a30a815a25d21e966ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:26:20.823128 13531 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c4a02e8a46a4a30a815a25d21e966ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:26:20.823335 13531 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c4a02e8a46a4a30a815a25d21e966ff [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:26:20.825718 13531 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c4a02e8a46a4a30a815a25d21e966ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c4a02e8a46a4a30a815a25d21e966ff" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 42979 } }
I20260811 22:26:20.826144 13531 leader_election.cc:302] T 00000000000000000000000000000000 P 0c4a02e8a46a4a30a815a25d21e966ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c4a02e8a46a4a30a815a25d21e966ff; no voters:
I20260811 22:26:20.826648 13531 leader_election.cc:288] T 00000000000000000000000000000000 P 0c4a02e8a46a4a30a815a25d21e966ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:26:20.826757 13538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c4a02e8a46a4a30a815a25d21e966ff [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:26:20.827440 13538 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c4a02e8a46a4a30a815a25d21e966ff [term 1 LEADER]: Becoming Leader. State: Replica: 0c4a02e8a46a4a30a815a25d21e966ff, State: Running, Role: LEADER
I20260811 22:26:20.827937 13538 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c4a02e8a46a4a30a815a25d21e966ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c4a02e8a46a4a30a815a25d21e966ff" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 42979 } }
I20260811 22:26:20.828509 13531 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c4a02e8a46a4a30a815a25d21e966ff [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:26:20.830521 13539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c4a02e8a46a4a30a815a25d21e966ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c4a02e8a46a4a30a815a25d21e966ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c4a02e8a46a4a30a815a25d21e966ff" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 42979 } } }
I20260811 22:26:20.830848 13541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c4a02e8a46a4a30a815a25d21e966ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c4a02e8a46a4a30a815a25d21e966ff. Latest consensus state: current_term: 1 leader_uuid: "0c4a02e8a46a4a30a815a25d21e966ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c4a02e8a46a4a30a815a25d21e966ff" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 42979 } } }
I20260811 22:26:20.831387 13539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c4a02e8a46a4a30a815a25d21e966ff [sys.catalog]: This master's current role is: LEADER
I20260811 22:26:20.831427 13541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c4a02e8a46a4a30a815a25d21e966ff [sys.catalog]: This master's current role is: LEADER
I20260811 22:26:20.840350 13552 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:26:20.845171 13552 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:26:20.853675 13552 catalog_manager.cc:1347] Generated new cluster ID: a550aebac2314181a9c069df8c07843d
I20260811 22:26:20.853775 13552 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:26:20.890174 13552 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:26:20.892043 13552 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:26:20.909435 13552 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c4a02e8a46a4a30a815a25d21e966ff: Generated new TSK 0
I20260811 22:26:20.910357 13552 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.106.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab.
Entry for principal kudu/127.29.106.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab.
I20260811 22:26:20.973269 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.1:0
--local_ip_for_outbound_sockets=127.29.106.1
--webserver_interface=127.29.106.1
--webserver_port=0
--tserver_master_addrs=127.29.106.62:42979
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab
--principal=kudu/127.29.106.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:34383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:26:21.297500 13569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:26:21.297811 13569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:26:21.297958 13569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:26:21.305554 13569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:26:21.305684 13569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:26:21.305837 13569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.1
I20260811 22:26:21.315678 13569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:34383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab
--principal=kudu/127.29.106.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.106.1
--webserver_port=0
--tserver_master_addrs=127.29.106.62:42979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.1
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:26:21.316915 13569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:26:21.318224 13569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:26:21.331629 13578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:21.331655 13583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:21.332043 13585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:26:21.332661 13569 server_base.cc:1034] running on GCE node
I20260811 22:26:21.333225 13569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:26:21.334100 13569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:26:21.335327 13569 hybrid_clock.cc:648] HybridClock initialized: now 1786487181335271 us; error 46 us; skew 500 ppm
I20260811 22:26:21.335624 13569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:26:21 dist-test-slave-gjrz krb5kdc[13378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487181, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:26:21.339591 13569 init.cc:377] Logged in from keytab as kudu/127.29.106.1@KRBTEST.COM (short username kudu)
I20260811 22:26:21.341426 13569 webserver.cc:460] Webserver started at http://127.29.106.1:33097/ using document root <none> and password file <none>
I20260811 22:26:21.342322 13569 fs_manager.cc:359] Metadata directory not provided
I20260811 22:26:21.342518 13569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:26:21.342962 13569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:26:21.346868 13569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-0/data/instance:
uuid: "4e6cbc1839b74e3c8b32e210df9a8d21"
format_stamp: "Formatted at 2026-08-11 22:26:21 on dist-test-slave-gjrz"
server_key: "557dcc78a672603bafa8f6d464c3c214"
server_key_iv: "dcd3ecd47b19cb55fbeec3d8a84391ab"
server_key_version: "encryptionkey@0"
I20260811 22:26:21.347740 13569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance:
uuid: "4e6cbc1839b74e3c8b32e210df9a8d21"
format_stamp: "Formatted at 2026-08-11 22:26:21 on dist-test-slave-gjrz"
server_key: "557dcc78a672603bafa8f6d464c3c214"
server_key_iv: "dcd3ecd47b19cb55fbeec3d8a84391ab"
server_key_version: "encryptionkey@0"
I20260811 22:26:21.352994 13569 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260811 22:26:21.356320 13597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:21.357564 13569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:26:21.357707 13569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-0/wal
uuid: "4e6cbc1839b74e3c8b32e210df9a8d21"
format_stamp: "Formatted at 2026-08-11 22:26:21 on dist-test-slave-gjrz"
server_key: "557dcc78a672603bafa8f6d464c3c214"
server_key_iv: "dcd3ecd47b19cb55fbeec3d8a84391ab"
server_key_version: "encryptionkey@0"
I20260811 22:26:21.357877 13569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:26:21.385422 13569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:26:21.389179 13569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:26:21.389441 13569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:26:21.390336 13569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:26:21.391645 13569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:26:21.391716 13569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:21.391834 13569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:26:21.391884 13569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:21.421084 13569 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.1:44601
I20260811 22:26:21.421152 13787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.1:44601 every 8 connection(s)
I20260811 22:26:21.422254 13569 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
I20260811 22:26:21.426613 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 13569
I20260811 22:26:21.426812 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance
I20260811 22:26:21.427476 30120 external_mini_cluster.cc:1428] Setting key 7f57e6528c584a118582dcfe4ee9e83e
Aug 11 22:26:21 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487181, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.1@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:26:21.442456 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:21.424627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:36161 (local address 127.29.106.62:42979)
0811 22:26:21.425041 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:21.425046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:21.425946 (+ 900us) server_negotiation.cc:407] Connection header received
0811 22:26:21.427927 (+ 1981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:21.427932 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:21.428010 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:21.428108 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:21.429866 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:21.430605 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:21.432762 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:21.433191 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:21.433287 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:21.437023 (+ 3736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:21.437028 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:21.437031 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:21.437064 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:21.439440 (+ 2376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:21.440262 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:21.440266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:21.440269 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:21.440392 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:21.440802 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:21.440805 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:21.440808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:21.441066 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:21.441200 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:21.441892 (+ 692us) server_negotiation.cc:299] Negotiation successful
0811 22:26:21.442121 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":209,"threads_started":1}
I20260811 22:26:21.443288 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:21.425127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.1:36161)
0811 22:26:21.425784 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:21.425822 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:21.427666 (+ 1844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:21.428366 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:21.428374 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:21.428887 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:21.429666 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:21.429694 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:21.430764 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:21.430769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:21.432631 (+ 1862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:21.432645 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:21.433396 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:21.433402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:21.433652 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:21.434292 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:21.434320 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:21.436815 (+ 2495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:26:21.439564 (+ 2749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:21.439573 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:21.439588 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:21.440135 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:21.440495 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:21.440500 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:21.440504 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:21.440685 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:21.441189 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:26:21.441196 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:26:21.441502 (+ 306us) client_negotiation.cc:769] Sending connection context
0811 22:26:21.441722 (+ 220us) client_negotiation.cc:240] Negotiation successful
0811 22:26:21.442223 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":257,"threads_started":1}
I20260811 22:26:21.444897 13788 heartbeater.cc:347] Connected to a master server at 127.29.106.62:42979
I20260811 22:26:21.445173 13788 heartbeater.cc:464] Registering TS with master...
I20260811 22:26:21.445819 13788 heartbeater.cc:511] Master 127.29.106.62:42979 requested a full tablet report, sending...
I20260811 22:26:21.447983 13459 ts_manager.cc:194] Registered new tserver with Master: 4e6cbc1839b74e3c8b32e210df9a8d21 (127.29.106.1:44601)
I20260811 22:26:21.449991 13459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.1@KRBTEST.COM'} at 127.29.106.1:36161
WARNING: no policy specified for kudu/127.29.106.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab.
Entry for principal kudu/127.29.106.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab.
I20260811 22:26:21.483320 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.2:0
--local_ip_for_outbound_sockets=127.29.106.2
--webserver_interface=127.29.106.2
--webserver_port=0
--tserver_master_addrs=127.29.106.62:42979
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab
--principal=kudu/127.29.106.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:34383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:26:21.809798 13798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:26:21.810051 13798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:26:21.810184 13798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:26:21.817294 13798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:26:21.817389 13798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:26:21.817533 13798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.2
I20260811 22:26:21.826776 13798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:34383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab
--principal=kudu/127.29.106.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.106.2
--webserver_port=0
--tserver_master_addrs=127.29.106.62:42979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.2
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:26:21.827929 13798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:26:21.829164 13798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:26:21.841261 13811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:21.841392 13809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:21.841427 13813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:26:21.842244 13798 server_base.cc:1034] running on GCE node
I20260811 22:26:21.842777 13798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:26:21.843645 13798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:26:21.844844 13798 hybrid_clock.cc:648] HybridClock initialized: now 1786487181844790 us; error 39 us; skew 500 ppm
I20260811 22:26:21.845127 13798 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:26:21 dist-test-slave-gjrz krb5kdc[13378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487181, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:26:21.848016 13798 init.cc:377] Logged in from keytab as kudu/127.29.106.2@KRBTEST.COM (short username kudu)
I20260811 22:26:21.849325 13798 webserver.cc:460] Webserver started at http://127.29.106.2:38411/ using document root <none> and password file <none>
I20260811 22:26:21.849967 13798 fs_manager.cc:359] Metadata directory not provided
I20260811 22:26:21.850082 13798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:26:21.850363 13798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:26:21.853078 13798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-1/data/instance:
uuid: "33096d0a5ccd49108fc8f027e96178b7"
format_stamp: "Formatted at 2026-08-11 22:26:21 on dist-test-slave-gjrz"
server_key: "15f48c4c01f1959187ef8ec63c0b9af9"
server_key_iv: "6aae7082f5866602e777028727ea378c"
server_key_version: "encryptionkey@0"
I20260811 22:26:21.853834 13798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance:
uuid: "33096d0a5ccd49108fc8f027e96178b7"
format_stamp: "Formatted at 2026-08-11 22:26:21 on dist-test-slave-gjrz"
server_key: "15f48c4c01f1959187ef8ec63c0b9af9"
server_key_iv: "6aae7082f5866602e777028727ea378c"
server_key_version: "encryptionkey@0"
I20260811 22:26:21.858835 13798 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260811 22:26:21.862207 13832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:21.863497 13798 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:26:21.863648 13798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-1/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-1/wal
uuid: "33096d0a5ccd49108fc8f027e96178b7"
format_stamp: "Formatted at 2026-08-11 22:26:21 on dist-test-slave-gjrz"
server_key: "15f48c4c01f1959187ef8ec63c0b9af9"
server_key_iv: "6aae7082f5866602e777028727ea378c"
server_key_version: "encryptionkey@0"
I20260811 22:26:21.863821 13798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:26:21.879546 13798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:26:21.882944 13798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:26:21.883186 13798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:26:21.884081 13798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:26:21.885291 13798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:26:21.885363 13798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:21.885470 13798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:26:21.885528 13798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:21.914729 13798 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.2:36315
I20260811 22:26:21.914790 14009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.2:36315 every 8 connection(s)
I20260811 22:26:21.915906 13798 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
I20260811 22:26:21.924417 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 13798
I20260811 22:26:21.924592 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance
I20260811 22:26:21.925009 30120 external_mini_cluster.cc:1428] Setting key 3fdea6662bdbbfbbadc5a4ec1621b0d3
Aug 11 22:26:21 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487181, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.2@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:26:21.935304 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:21.918286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:58531 (local address 127.29.106.62:42979)
0811 22:26:21.918505 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:21.918511 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:21.919650 (+ 1139us) server_negotiation.cc:407] Connection header received
0811 22:26:21.921368 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:21.921373 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:21.921445 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:21.921538 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:21.922951 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:21.923701 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:21.925260 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:21.925841 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:21.925978 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:21.929555 (+ 3577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:21.929561 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:21.929565 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:21.929602 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:21.932448 (+ 2846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:21.933150 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:21.933155 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:21.933158 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:21.933314 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:21.933670 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:21.933674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:21.933676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:21.933975 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:21.934124 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:21.934748 (+ 624us) server_negotiation.cc:299] Negotiation successful
0811 22:26:21.934956 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260811 22:26:21.936002 14016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:21.918750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.2:58531)
0811 22:26:21.919486 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:21.919525 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:21.921120 (+ 1595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:21.921731 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:21.921742 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:21.922259 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:21.922788 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:21.922806 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:21.923856 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:21.923860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:21.925119 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:21.925133 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:21.926052 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:21.926060 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:21.926334 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:21.927034 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:21.927071 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:21.929344 (+ 2273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:26:21.932568 (+ 3224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:21.932574 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:21.932586 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:21.933038 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:21.933398 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:21.933401 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:21.933404 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:21.933576 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:21.934116 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:26:21.934122 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:26:21.934404 (+ 282us) client_negotiation.cc:769] Sending connection context
0811 22:26:21.934626 (+ 222us) client_negotiation.cc:240] Negotiation successful
0811 22:26:21.934961 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":269,"threads_started":1}
I20260811 22:26:21.937220 14011 heartbeater.cc:347] Connected to a master server at 127.29.106.62:42979
I20260811 22:26:21.937525 14011 heartbeater.cc:464] Registering TS with master...
I20260811 22:26:21.938256 14011 heartbeater.cc:511] Master 127.29.106.62:42979 requested a full tablet report, sending...
I20260811 22:26:21.939692 13459 ts_manager.cc:194] Registered new tserver with Master: 33096d0a5ccd49108fc8f027e96178b7 (127.29.106.2:36315)
I20260811 22:26:21.940730 13459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.2@KRBTEST.COM'} at 127.29.106.2:58531
WARNING: no policy specified for kudu/127.29.106.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab.
Entry for principal kudu/127.29.106.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab.
I20260811 22:26:21.976298 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.3:0
--local_ip_for_outbound_sockets=127.29.106.3
--webserver_interface=127.29.106.3
--webserver_port=0
--tserver_master_addrs=127.29.106.62:42979
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab
--principal=kudu/127.29.106.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:34383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:26:22.291196 14023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:26:22.291451 14023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:26:22.291569 14023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:26:22.298440 14023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:26:22.298530 14023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:26:22.298683 14023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.3
I20260811 22:26:22.308202 14023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:34383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab
--principal=kudu/127.29.106.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.106.3
--webserver_port=0
--tserver_master_addrs=127.29.106.62:42979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.3
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:26:22.309311 14023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:26:22.310549 14023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:26:22.323205 14034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:22.323115 14038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:22.323177 14036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:26:22.324036 14023 server_base.cc:1034] running on GCE node
I20260811 22:26:22.324571 14023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:26:22.325419 14023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:26:22.326608 14023 hybrid_clock.cc:648] HybridClock initialized: now 1786487182326578 us; error 32 us; skew 500 ppm
I20260811 22:26:22.326877 14023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:26:22 dist-test-slave-gjrz krb5kdc[13378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487182, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:26:22.329970 14023 init.cc:377] Logged in from keytab as kudu/127.29.106.3@KRBTEST.COM (short username kudu)
I20260811 22:26:22.331250 14023 webserver.cc:460] Webserver started at http://127.29.106.3:42921/ using document root <none> and password file <none>
I20260811 22:26:22.331888 14023 fs_manager.cc:359] Metadata directory not provided
I20260811 22:26:22.332005 14023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:26:22.332278 14023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:26:22.335009 14023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-2/data/instance:
uuid: "b3ae54365e9448cc864a4f6199b19c53"
format_stamp: "Formatted at 2026-08-11 22:26:22 on dist-test-slave-gjrz"
server_key: "41b4093f5439691400e933a765cb87a3"
server_key_iv: "6357b915d941627c985df7c2924bd31d"
server_key_version: "encryptionkey@0"
I20260811 22:26:22.335788 14023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance:
uuid: "b3ae54365e9448cc864a4f6199b19c53"
format_stamp: "Formatted at 2026-08-11 22:26:22 on dist-test-slave-gjrz"
server_key: "41b4093f5439691400e933a765cb87a3"
server_key_iv: "6357b915d941627c985df7c2924bd31d"
server_key_version: "encryptionkey@0"
I20260811 22:26:22.340507 14023 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 22:26:22.343663 14047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:22.344848 14023 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260811 22:26:22.344992 14023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-2/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-2/wal
uuid: "b3ae54365e9448cc864a4f6199b19c53"
format_stamp: "Formatted at 2026-08-11 22:26:22 on dist-test-slave-gjrz"
server_key: "41b4093f5439691400e933a765cb87a3"
server_key_iv: "6357b915d941627c985df7c2924bd31d"
server_key_version: "encryptionkey@0"
I20260811 22:26:22.345171 14023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:26:22.363441 14023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:26:22.367033 14023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:26:22.367254 14023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:26:22.368113 14023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:26:22.369415 14023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:26:22.369488 14023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:22.369587 14023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:26:22.369629 14023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:22.397133 14023 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.3:43935
I20260811 22:26:22.397207 14252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.3:43935 every 8 connection(s)
I20260811 22:26:22.398295 14023 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
I20260811 22:26:22.405542 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 14023
I20260811 22:26:22.405723 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance
I20260811 22:26:22.406126 30120 external_mini_cluster.cc:1428] Setting key 6b9e23157e13433e2ac3198d4fe1ad89
Aug 11 22:26:22 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487182, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.3@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:26:22.416720 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:22.400680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:46617 (local address 127.29.106.62:42979)
0811 22:26:22.400861 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:22.400866 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:22.401919 (+ 1053us) server_negotiation.cc:407] Connection header received
0811 22:26:22.403736 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:22.403742 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:22.403832 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:22.403921 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:22.405328 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:22.406159 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:22.408070 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:22.408494 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:22.408607 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:22.411797 (+ 3190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:22.411802 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:22.411806 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:22.411838 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:22.414234 (+ 2396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:22.414863 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:22.414867 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:22.414869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:22.414988 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:22.415356 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:22.415359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:22.415361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:22.415616 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:22.415729 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:22.416312 (+ 583us) server_negotiation.cc:299] Negotiation successful
0811 22:26:22.416468 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260811 22:26:22.417608 14259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:22.401084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.3:46617)
0811 22:26:22.401759 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:22.401805 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:22.403471 (+ 1666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:22.404158 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:22.404166 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:22.404641 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:22.405156 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:22.405173 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:22.406323 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:22.406329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:22.407922 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:22.407934 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:22.408712 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:22.408720 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:22.408966 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:22.409562 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:22.409589 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:22.411604 (+ 2015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:26:22.414368 (+ 2764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:22.414375 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:22.414385 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:22.414772 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:22.415067 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:22.415070 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:22.415072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:22.415239 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:22.415733 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:26:22.415739 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:26:22.416018 (+ 279us) client_negotiation.cc:769] Sending connection context
0811 22:26:22.416222 (+ 204us) client_negotiation.cc:240] Negotiation successful
0811 22:26:22.416551 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":262,"threads_started":1}
I20260811 22:26:22.418816 14254 heartbeater.cc:347] Connected to a master server at 127.29.106.62:42979
I20260811 22:26:22.419095 14254 heartbeater.cc:464] Registering TS with master...
I20260811 22:26:22.419849 14254 heartbeater.cc:511] Master 127.29.106.62:42979 requested a full tablet report, sending...
I20260811 22:26:22.421305 13459 ts_manager.cc:194] Registered new tserver with Master: b3ae54365e9448cc864a4f6199b19c53 (127.29.106.3:43935)
I20260811 22:26:22.422098 13459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.3@KRBTEST.COM'} at 127.29.106.3:46617
I20260811 22:26:22.433019 30120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 22:26:22.452872 13788 heartbeater.cc:503] Master 127.29.106.62:42979 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.29.106.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.106.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.106.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.1.keytab.
Entry for principal oryx/127.29.106.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.1.keytab.
WARNING: no policy specified for oryx/127.29.106.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.106.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.106.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.2.keytab.
Entry for principal oryx/127.29.106.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.2.keytab.
WARNING: no policy specified for oryx/127.29.106.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.106.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.106.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.3.keytab.
Entry for principal oryx/127.29.106.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.3.keytab.
I20260811 22:26:22.571796 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 13569
Aug 11 22:26:22 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.1@KRBTEST.COM
I20260811 22:26:22.586512 14271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:22.572800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53252 (local address 127.29.106.1:44601)
0811 22:26:22.573409 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:22.573417 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:22.573446 (+ 29us) server_negotiation.cc:407] Connection header received
0811 22:26:22.573511 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:22.573518 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:22.573744 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:22.573888 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:22.575161 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:22.576228 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:22.577405 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:22.577954 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:22.578056 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:22.581675 (+ 3619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:22.581689 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:22.581704 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:22.581740 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:22.584302 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:22.584871 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:22.584877 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:22.584879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:22.585002 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:22.585289 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:22.585292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:22.585294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:22.585613 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:22.585833 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:22.586018 (+ 185us) server_negotiation.cc:299] Negotiation successful
0811 22:26:22.586228 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":287,"threads_started":1}
I20260811 22:26:22.587781 13781 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:26:22.672235 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 13569
I20260811 22:26:22.695173 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 13798
Aug 11 22:26:22 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.2@KRBTEST.COM
I20260811 22:26:22.710177 14274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:22.696142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59508 (local address 127.29.106.2:36315)
0811 22:26:22.696667 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:22.696674 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:22.696699 (+ 25us) server_negotiation.cc:407] Connection header received
0811 22:26:22.696766 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:22.696772 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:22.696975 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:22.697124 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:22.698352 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:22.699429 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:22.700666 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:22.701202 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:22.701314 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:22.705296 (+ 3982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:22.705307 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:22.705322 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:22.705354 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:22.707991 (+ 2637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:22.708586 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:22.708592 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:22.708594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:22.708708 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:22.708977 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:22.708979 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:22.708981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:22.709288 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:22.709512 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:22.709716 (+ 204us) server_negotiation.cc:299] Negotiation successful
0811 22:26:22.709901 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":269,"threads_started":1}
I20260811 22:26:22.711449 14003 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:26:22.792474 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 13798
I20260811 22:26:22.814388 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 14023
Aug 11 22:26:22 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.3@KRBTEST.COM
I20260811 22:26:22.829780 14278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:22.815335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44892 (local address 127.29.106.3:43935)
0811 22:26:22.815957 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:22.815964 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:22.815991 (+ 27us) server_negotiation.cc:407] Connection header received
0811 22:26:22.816071 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:22.816077 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:22.816283 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:22.816410 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:22.817672 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:22.818693 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:22.820018 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:22.820567 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:22.820659 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:22.824740 (+ 4081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:22.824751 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:22.824765 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:22.824801 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:22.827237 (+ 2436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:22.827931 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:22.827938 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:22.827940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:22.828077 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:22.828367 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:22.828370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:22.828372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:22.828714 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:22.829004 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:22.829280 (+ 276us) server_negotiation.cc:299] Negotiation successful
0811 22:26:22.829498 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":315,"threads_started":1}
I20260811 22:26:22.831001 14244 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:26:22.911655 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 14023
I20260811 22:26:22.934508 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 13403
I20260811 22:26:22.935628 13524 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:26:23.037482 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 13403
I20260811 22:26:23.063467 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:42979
--webserver_interface=127.29.106.62
--webserver_port=36249
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:34383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:42979
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:26:23.392053 14286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:26:23.392318 14286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:26:23.392416 14286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:26:23.399434 14286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:26:23.399533 14286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:26:23.399576 14286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:26:23.399631 14286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:26:23.399673 14286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:26:23.409224 14286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:34383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:42979
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:42979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.106.62
--webserver_port=36249
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:26:23.410383 14286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:26:23.411619 14286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:26:23.422752 14296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:23.422788 14294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:23.423228 14293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:26:23.423676 14286 server_base.cc:1034] running on GCE node
I20260811 22:26:23.424401 14286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:26:23.425729 14286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:26:23.426934 14286 hybrid_clock.cc:648] HybridClock initialized: now 1786487183426881 us; error 41 us; skew 500 ppm
I20260811 22:26:23.427256 14286 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:26:23 dist-test-slave-gjrz krb5kdc[13378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487183, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:26:23.430191 14286 init.cc:377] Logged in from keytab as kudu/127.29.106.62@KRBTEST.COM (short username kudu)
I20260811 22:26:23.431628 14286 webserver.cc:460] Webserver started at http://127.29.106.62:36249/ using document root <none> and password file <none>
I20260811 22:26:23.432267 14286 fs_manager.cc:359] Metadata directory not provided
I20260811 22:26:23.432395 14286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:26:23.437487 14286 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20260811 22:26:23.440128 14311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:23.441324 14286 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260811 22:26:23.441469 14286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "0c4a02e8a46a4a30a815a25d21e966ff"
format_stamp: "Formatted at 2026-08-11 22:26:20 on dist-test-slave-gjrz"
server_key: "1159e82266cfda43355ae2fa9362a535"
server_key_iv: "3a3a244bbd1add8c9aad96758e6103cb"
server_key_version: "encryptionkey@0"
I20260811 22:26:23.442414 14286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:26:23.458240 14286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:26:23.461787 14286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:26:23.462045 14286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:26:23.479070 14286 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.62:42979
I20260811 22:26:23.479091 14399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.62:42979 every 8 connection(s)
I20260811 22:26:23.480509 14286 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:26:23.487030 14401 sys_catalog.cc:263] Verifying existing consensus state
I20260811 22:26:23.487701 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 14286
I20260811 22:26:23.488656 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.1:44601
--local_ip_for_outbound_sockets=127.29.106.1
--tserver_master_addrs=127.29.106.62:42979
--webserver_port=33097
--webserver_interface=127.29.106.1
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab
--principal=kudu/127.29.106.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:34383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.29.106.1
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 22:26:23.491468 14401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c4a02e8a46a4a30a815a25d21e966ff: Bootstrap starting.
I20260811 22:26:23.511991 14401 log.cc:824] T 00000000000000000000000000000000 P 0c4a02e8a46a4a30a815a25d21e966ff: Log is configured to *not* fsync() on all Append() calls
I20260811 22:26:23.522646 14401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c4a02e8a46a4a30a815a25d21e966ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 22:26:23.523135 14401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c4a02e8a46a4a30a815a25d21e966ff: Bootstrap complete.
I20260811 22:26:23.532042 14401 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c4a02e8a46a4a30a815a25d21e966ff [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c4a02e8a46a4a30a815a25d21e966ff" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 42979 } }
I20260811 22:26:23.532850 14401 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c4a02e8a46a4a30a815a25d21e966ff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c4a02e8a46a4a30a815a25d21e966ff, State: Initialized, Role: FOLLOWER
I20260811 22:26:23.533397 14401 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c4a02e8a46a4a30a815a25d21e966ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c4a02e8a46a4a30a815a25d21e966ff" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 42979 } }
I20260811 22:26:23.533624 14401 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c4a02e8a46a4a30a815a25d21e966ff [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:26:23.533728 14401 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c4a02e8a46a4a30a815a25d21e966ff [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:26:23.533882 14401 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c4a02e8a46a4a30a815a25d21e966ff [term 1 FOLLOWER]: Advancing to term 2
I20260811 22:26:23.537472 14401 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c4a02e8a46a4a30a815a25d21e966ff [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c4a02e8a46a4a30a815a25d21e966ff" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 42979 } }
I20260811 22:26:23.537909 14401 leader_election.cc:302] T 00000000000000000000000000000000 P 0c4a02e8a46a4a30a815a25d21e966ff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c4a02e8a46a4a30a815a25d21e966ff; no voters:
I20260811 22:26:23.538493 14401 leader_election.cc:288] T 00000000000000000000000000000000 P 0c4a02e8a46a4a30a815a25d21e966ff [CANDIDATE]: Term 2 election: Requested vote from peers
I20260811 22:26:23.538633 14407 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c4a02e8a46a4a30a815a25d21e966ff [term 2 FOLLOWER]: Leader election won for term 2
I20260811 22:26:23.539752 14407 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c4a02e8a46a4a30a815a25d21e966ff [term 2 LEADER]: Becoming Leader. State: Replica: 0c4a02e8a46a4a30a815a25d21e966ff, State: Running, Role: LEADER
I20260811 22:26:23.540292 14407 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c4a02e8a46a4a30a815a25d21e966ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c4a02e8a46a4a30a815a25d21e966ff" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 42979 } }
I20260811 22:26:23.540587 14401 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c4a02e8a46a4a30a815a25d21e966ff [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:26:23.542692 14409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c4a02e8a46a4a30a815a25d21e966ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0c4a02e8a46a4a30a815a25d21e966ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c4a02e8a46a4a30a815a25d21e966ff" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 42979 } } }
I20260811 22:26:23.543041 14409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c4a02e8a46a4a30a815a25d21e966ff [sys.catalog]: This master's current role is: LEADER
I20260811 22:26:23.544514 14412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c4a02e8a46a4a30a815a25d21e966ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c4a02e8a46a4a30a815a25d21e966ff. Latest consensus state: current_term: 2 leader_uuid: "0c4a02e8a46a4a30a815a25d21e966ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c4a02e8a46a4a30a815a25d21e966ff" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 42979 } } }
I20260811 22:26:23.544809 14412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c4a02e8a46a4a30a815a25d21e966ff [sys.catalog]: This master's current role is: LEADER
I20260811 22:26:23.545890 14415 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:26:23.555310 14415 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:26:23.556789 14415 catalog_manager.cc:1259] Loaded cluster ID: a550aebac2314181a9c069df8c07843d
I20260811 22:26:23.556927 14415 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:26:23.561028 14415 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:26:23.562554 14415 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0c4a02e8a46a4a30a815a25d21e966ff: Loaded TSK: 0
I20260811 22:26:23.564625 14415 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260811 22:26:23.853178 14404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:26:23.853441 14404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:26:23.853564 14404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:26:23.861136 14404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:26:23.861238 14404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:26:23.861402 14404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.1
I20260811 22:26:23.871019 14404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:34383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.1.keytab
--principal=oryx/127.29.106.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.1:44601
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.106.1
--webserver_port=33097
--tserver_master_addrs=127.29.106.62:42979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.1
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:26:23.872241 14404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:26:23.873446 14404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:26:23.887605 14446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:23.887634 14448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:23.888339 14458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:26:23.888659 14404 server_base.cc:1034] running on GCE node
I20260811 22:26:23.889351 14404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:26:23.890422 14404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:26:23.891661 14404 hybrid_clock.cc:648] HybridClock initialized: now 1786487183891602 us; error 44 us; skew 500 ppm
I20260811 22:26:23.892019 14404 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:26:23 dist-test-slave-gjrz krb5kdc[13378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487183, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.106.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:26:23.900733 14404 init.cc:377] Logged in from keytab as oryx/127.29.106.1@KRBTEST.COM (short username oryx)
I20260811 22:26:23.902387 14404 webserver.cc:460] Webserver started at http://127.29.106.1:33097/ using document root <none> and password file <none>
I20260811 22:26:23.903298 14404 fs_manager.cc:359] Metadata directory not provided
I20260811 22:26:23.903443 14404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:26:23.910100 14404 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.006s
I20260811 22:26:23.913386 14474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:23.914932 14404 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 22:26:23.915109 14404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-0/wal
uuid: "4e6cbc1839b74e3c8b32e210df9a8d21"
format_stamp: "Formatted at 2026-08-11 22:26:21 on dist-test-slave-gjrz"
server_key: "557dcc78a672603bafa8f6d464c3c214"
server_key_iv: "dcd3ecd47b19cb55fbeec3d8a84391ab"
server_key_version: "encryptionkey@0"
I20260811 22:26:23.916460 14404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:26:23.946456 14404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:26:23.951205 14404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:26:23.951516 14404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:26:23.952450 14404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:26:23.953774 14404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:26:23.953877 14404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:23.953976 14404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:26:23.954046 14404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:23.983112 14404 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.1:44601
I20260811 22:26:23.983182 14674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.1:44601 every 8 connection(s)
I20260811 22:26:23.984479 14404 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
I20260811 22:26:23.985493 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 14404
I20260811 22:26:23.986102 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.2:36315
--local_ip_for_outbound_sockets=127.29.106.2
--tserver_master_addrs=127.29.106.62:42979
--webserver_port=38411
--webserver_interface=127.29.106.2
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab
--principal=kudu/127.29.106.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:34383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.29.106.2
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 22:26:23 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.1@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:23 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.1@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
W20260811 22:26:24.309656 14682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:26:24.309919 14682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:26:24.310039 14682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:26:24.317066 14682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:26:24.317170 14682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:26:24.317343 14682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.2
I20260811 22:26:24.327008 14682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:34383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.2.keytab
--principal=oryx/127.29.106.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.2:36315
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.106.2
--webserver_port=38411
--tserver_master_addrs=127.29.106.62:42979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.2
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:26:24.328245 14682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:26:24.329520 14682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:26:24.351172 14695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:24.351238 14700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:24.351150 14692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:26:24.352787 14682 server_base.cc:1034] running on GCE node
I20260811 22:26:24.353600 14682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:26:24.354893 14682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:26:24.356137 14682 hybrid_clock.cc:648] HybridClock initialized: now 1786487184356076 us; error 43 us; skew 500 ppm
I20260811 22:26:24.356547 14682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:26:24 dist-test-slave-gjrz krb5kdc[13378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487184, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.106.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:26:24.360900 14682 init.cc:377] Logged in from keytab as oryx/127.29.106.2@KRBTEST.COM (short username oryx)
I20260811 22:26:24.362725 14682 webserver.cc:460] Webserver started at http://127.29.106.2:38411/ using document root <none> and password file <none>
I20260811 22:26:24.363723 14682 fs_manager.cc:359] Metadata directory not provided
I20260811 22:26:24.363885 14682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:26:24.371389 14682 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.008s sys 0.000s
I20260811 22:26:24.374864 14710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:24.376287 14682 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:26:24.376449 14682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-1/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-1/wal
uuid: "33096d0a5ccd49108fc8f027e96178b7"
format_stamp: "Formatted at 2026-08-11 22:26:21 on dist-test-slave-gjrz"
server_key: "15f48c4c01f1959187ef8ec63c0b9af9"
server_key_iv: "6aae7082f5866602e777028727ea378c"
server_key_version: "encryptionkey@0"
I20260811 22:26:24.377404 14682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:26:24.392688 14682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:26:24.397204 14682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:26:24.397517 14682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:26:24.398681 14682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:26:24.400494 14682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:26:24.400616 14682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:24.400753 14682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:26:24.400837 14682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:24.433286 14682 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.2:36315
I20260811 22:26:24.433305 14913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.2:36315 every 8 connection(s)
I20260811 22:26:24.435320 14682 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
I20260811 22:26:24.441772 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 14682
I20260811 22:26:24.442525 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.3:43935
--local_ip_for_outbound_sockets=127.29.106.3
--tserver_master_addrs=127.29.106.62:42979
--webserver_port=42921
--webserver_interface=127.29.106.3
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab
--principal=kudu/127.29.106.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:34383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.29.106.3
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 22:26:24 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.2@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:24 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.2@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
W20260811 22:26:24.777626 14933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:26:24.777886 14933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:26:24.778039 14933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:26:24.785622 14933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:26:24.785728 14933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:26:24.785882 14933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.3
I20260811 22:26:24.795073 14933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:34383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/krb5kdc/oryx_127.29.106.3.keytab
--principal=oryx/127.29.106.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.3:43935
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.106.3
--webserver_port=42921
--tserver_master_addrs=127.29.106.62:42979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.3
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:26:24.796240 14933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:26:24.797425 14933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:26:24.811646 14949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:24.812206 14948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:24.832799 14952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:26:24.833171 14933 server_base.cc:1034] running on GCE node
I20260811 22:26:24.833724 14933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:26:24.834661 14933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:26:24.835893 14933 hybrid_clock.cc:648] HybridClock initialized: now 1786487184835837 us; error 49 us; skew 500 ppm
I20260811 22:26:24.836198 14933 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:26:24 dist-test-slave-gjrz krb5kdc[13378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487184, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.106.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:26:24.839329 14933 init.cc:377] Logged in from keytab as oryx/127.29.106.3@KRBTEST.COM (short username oryx)
I20260811 22:26:24.840902 14933 webserver.cc:460] Webserver started at http://127.29.106.3:42921/ using document root <none> and password file <none>
I20260811 22:26:24.841610 14933 fs_manager.cc:359] Metadata directory not provided
I20260811 22:26:24.841774 14933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:26:24.846952 14933 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260811 22:26:24.849557 14970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:24.850787 14933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 22:26:24.850935 14933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-2/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-2/wal
uuid: "b3ae54365e9448cc864a4f6199b19c53"
format_stamp: "Formatted at 2026-08-11 22:26:22 on dist-test-slave-gjrz"
server_key: "41b4093f5439691400e933a765cb87a3"
server_key_iv: "6357b915d941627c985df7c2924bd31d"
server_key_version: "encryptionkey@0"
I20260811 22:26:24.851890 14933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:26:24.871853 14933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:26:24.876093 14933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:26:24.876323 14933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:26:24.877257 14933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:26:24.878551 14933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:26:24.878645 14933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:24.878747 14933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:26:24.878816 14933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:24.921449 14933 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.3:43935
I20260811 22:26:24.921507 15168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.3:43935 every 8 connection(s)
I20260811 22:26:24.923048 14933 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
I20260811 22:26:24.933212 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 14933
Aug 11 22:26:24 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.3@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:24 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.3@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:26:24.969193 15173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:24.934200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54342 (local address 127.29.106.62:42979)
0811 22:26:24.946747 (+ 12547us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:24.946753 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:24.946777 (+ 24us) server_negotiation.cc:407] Connection header received
0811 22:26:24.946855 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:24.946860 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:24.946954 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:24.947097 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:24.949308 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:24.950606 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:24.952642 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:24.953458 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:24.953604 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:24.957608 (+ 4004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:24.957620 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:24.957642 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:24.957709 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:24.963397 (+ 5688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:24.964321 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:24.964330 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:24.964340 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:24.964556 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:24.965077 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:24.965082 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:24.965086 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:24.965731 (+ 645us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:24.967302 (+ 1571us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:24.967593 (+ 291us) server_negotiation.cc:299] Negotiation successful
0811 22:26:24.968087 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":12308,"thread_start_us":5205,"threads_started":1}
W20260811 22:26:25.241048 14680 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.244s user 0.000s sys 0.014s
W20260811 22:26:25.243396 14681 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:23.987421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:42759 (local address 127.29.106.62:42979)
0811 22:26:23.989150 (+ 1729us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:23.989168 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:23.989227 (+ 59us) server_negotiation.cc:407] Connection header received
0811 22:26:23.990743 (+ 1516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:23.990757 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:23.991061 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:23.991419 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:23.992869 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:23.994082 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:23.995616 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:23.996244 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:23.996424 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:25.242814 (+1246390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.106.1:42759: BlockingRecv error: recv got EOF from 127.29.106.1:42759 (error 108)
Metrics: {"server-negotiator.queue_time_us":1356,"thread_start_us":1102,"threads_started":1}
I20260811 22:26:25.243571 14680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:23.987408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.1:42759)
0811 22:26:23.988178 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:23.988222 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:23.990203 (+ 1981us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:23.991580 (+ 1377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:23.991591 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:23.992179 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:23.992679 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:23.992698 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:23.994242 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:23.994246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:23.995462 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:23.995476 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:23.996464 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:23.996471 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:23.996721 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:23.997298 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:23.997319 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:25.242161 (+1244842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":291,"threads_started":1}
W20260811 22:26:25.245337 14675 heartbeater.cc:644] Failed to heartbeat to 127.29.106.62:42979 (0 consecutive failures): Not authorized: Failed to ping master at 127.29.106.62:42979: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Aug 11 22:26:25 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.1@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:25 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.1@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
W20260811 22:26:25.264657 14681 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:25.246431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:53083 (local address 127.29.106.62:42979)
0811 22:26:25.246653 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:25.246660 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:25.246684 (+ 24us) server_negotiation.cc:407] Connection header received
0811 22:26:25.246900 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:25.246906 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:25.246996 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:25.247071 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:25.248333 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:25.249775 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:25.252689 (+ 2914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:25.253633 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:25.253760 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:25.264356 (+ 10596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.106.1:53083: BlockingRecv error: recv got EOF from 127.29.106.1:53083 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260811 22:26:25.265352 14680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:25.246187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.1:53083)
0811 22:26:25.246576 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:25.246600 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:25.246728 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:25.247351 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:25.247357 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:25.247899 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:25.248148 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:25.248162 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:25.250000 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:25.250006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:25.252398 (+ 2392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:25.252419 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:25.254035 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:25.254042 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:25.254240 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:25.255070 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:25.255097 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:25.265045 (+ 9948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238}
Aug 11 22:26:25 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.1@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:25 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.1@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:26:25.278918 14680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:25.266259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.1:36753)
0811 22:26:25.266582 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:25.266603 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:25.266757 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:25.267335 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:25.267341 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:25.267790 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:25.268001 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:25.268013 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:25.269615 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:25.269622 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:25.271283 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:25.271301 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:25.272575 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:25.272582 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:25.272797 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:25.273465 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:25.273490 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:25.278614 (+ 5124us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204}
W20260811 22:26:25.279139 14681 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:25.266527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:36753 (local address 127.29.106.62:42979)
0811 22:26:25.266696 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:25.266702 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:25.266722 (+ 20us) server_negotiation.cc:407] Connection header received
0811 22:26:25.266904 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:25.266909 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:25.267009 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:25.267082 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:25.268174 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:25.269437 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:25.271489 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:25.272206 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:25.272303 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:25.278848 (+ 6545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.106.1:36753: BlockingRecv error: recv got EOF from 127.29.106.1:36753 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260811 22:26:25.279668 14675 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260811 22:26:25.601226 14930 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.151s user 0.007s sys 0.010s
W20260811 22:26:25.602684 14924 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:24.438169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:44261 (local address 127.29.106.62:42979)
0811 22:26:24.438598 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:24.438603 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:24.439877 (+ 1274us) server_negotiation.cc:407] Connection header received
0811 22:26:24.442046 (+ 2169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:24.442054 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:24.442157 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:24.442282 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:24.444281 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:24.445559 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:24.447717 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:24.448430 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:24.448567 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:25.602456 (+1153889us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.106.2:44261: BlockingRecv error: recv got EOF from 127.29.106.2:44261 (error 108)
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":223,"threads_started":1}
I20260811 22:26:25.603756 14930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:24.438893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.2:44261)
0811 22:26:24.439715 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:24.439757 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:24.441687 (+ 1930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:24.442571 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:24.442583 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:24.443289 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:24.444072 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:24.444096 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:24.445745 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:24.445749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:24.447579 (+ 1830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:24.447593 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:24.448669 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:24.448676 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:24.448945 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:24.449646 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:24.449683 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:25.602325 (+1152642us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":361,"threads_started":1}
W20260811 22:26:25.604713 14917 heartbeater.cc:644] Failed to heartbeat to 127.29.106.62:42979 (0 consecutive failures): Not authorized: Failed to ping master at 127.29.106.62:42979: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Aug 11 22:26:25 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.2@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:25 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.2@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:26:25.615240 14930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:25.605299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.2:52783)
0811 22:26:25.605531 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:25.605556 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:25.605701 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:25.605997 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:25.606002 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:25.606564 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:25.606804 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:25.606816 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:25.607860 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:25.607866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:25.609586 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:25.609604 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:25.610305 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:25.610311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:25.610518 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:25.611154 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:25.611179 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:25.614979 (+ 3800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":63}
W20260811 22:26:25.615427 14924 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:25.605397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:52783 (local address 127.29.106.62:42979)
0811 22:26:25.605528 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:25.605532 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:25.605624 (+ 92us) server_negotiation.cc:407] Connection header received
0811 22:26:25.605806 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:25.605810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:25.605869 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:25.605954 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:25.606946 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:25.607739 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:25.609731 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:25.610136 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:25.610224 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:25.615174 (+ 4950us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.106.2:52783: BlockingRecv error: recv got EOF from 127.29.106.2:52783 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 11 22:26:25 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.2@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:25 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.2@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:26:25.624794 15180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:25.615818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.2:44177)
0811 22:26:25.616354 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:25.616370 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:25.616465 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:25.616757 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:25.616760 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:25.617154 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:25.617357 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:25.617364 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:25.618788 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:25.618792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:25.619876 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:25.619892 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:25.620722 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:25.620727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:25.620885 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:25.621454 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:25.621476 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:25.624523 (+ 3047us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":237,"threads_started":1}
W20260811 22:26:25.624941 14924 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:25.616083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:44177 (local address 127.29.106.62:42979)
0811 22:26:25.616262 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:25.616269 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:25.616437 (+ 168us) server_negotiation.cc:407] Connection header received
0811 22:26:25.616546 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:25.616552 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:25.616640 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:25.616752 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:25.617490 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:25.618673 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:25.620021 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:25.620561 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:25.620668 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:25.624713 (+ 4045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.106.2:44177: BlockingRecv error: recv got EOF from 127.29.106.2:44177 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260811 22:26:25.625151 14917 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260811 22:26:26.016939 15175 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.057s user 0.007s sys 0.014s
W20260811 22:26:26.018251 15177 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:24.925821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:34447 (local address 127.29.106.62:42979)
0811 22:26:24.939665 (+ 13844us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:24.939672 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:24.948069 (+ 8397us) server_negotiation.cc:407] Connection header received
0811 22:26:24.950649 (+ 2580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:24.950654 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:24.950725 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:24.950790 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:24.953167 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:24.954678 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:24.957308 (+ 2630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:24.958040 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:24.958141 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:26.017992 (+1059851us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.106.3:34447: BlockingRecv error: recv got EOF from 127.29.106.3:34447 (error 108)
Metrics: {"mutex_wait_us":98,"server-negotiator.queue_time_us":13713,"thread_start_us":20805,"threads_started":1}
I20260811 22:26:26.018886 15175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:24.926430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.3:34447)
0811 22:26:24.947912 (+ 21482us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:24.947967 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:24.950371 (+ 2404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:24.951242 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:24.951250 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:24.952064 (+ 814us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:24.952924 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:24.952961 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:24.954884 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:24.954890 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:24.957110 (+ 2220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:24.957128 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:24.958591 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:24.958599 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:24.958951 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:24.959643 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:24.959680 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:26.017818 (+1058138us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":21126,"thread_start_us":20911,"threads_started":1}
W20260811 22:26:26.019790 15169 heartbeater.cc:644] Failed to heartbeat to 127.29.106.62:42979 (0 consecutive failures): Not authorized: Failed to ping master at 127.29.106.62:42979: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Aug 11 22:26:26 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.3@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:26 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.3@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:26:26.028527 15175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:26.020401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.3:52201)
0811 22:26:26.020588 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:26.020605 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:26.020694 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:26.020954 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:26.020957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:26.021376 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:26.021544 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:26.021554 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:26.022602 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:26.022607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:26.023666 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:26.023680 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:26.024354 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:26.024359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:26.024477 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:26.025037 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:26.025056 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:26.028302 (+ 3246us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":60}
W20260811 22:26:26.028748 15177 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:26.020464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:52201 (local address 127.29.106.62:42979)
0811 22:26:26.020614 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:26.020618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:26.020638 (+ 20us) server_negotiation.cc:407] Connection header received
0811 22:26:26.020796 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:26.020799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:26.020859 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:26.020929 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:26.021661 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:26.022450 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:26.023798 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:26.024203 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:26.024295 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:26.028510 (+ 4215us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.106.3:52201: BlockingRecv error: recv got EOF from 127.29.106.3:52201 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 11 22:26:26 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.3@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:26 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.3@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:26:26.036429 15175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:26.029421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.3:46805)
0811 22:26:26.029570 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:26.029598 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:26.029694 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:26.029986 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:26.029989 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:26.030266 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:26.030406 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:26.030413 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:26.031349 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:26.031354 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:26.032266 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:26.032277 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:26.032906 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:26.032910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:26.033025 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:26.033469 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:26.033483 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:26.036256 (+ 2773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20260811 22:26:26.036662 15177 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:26.029500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:46805 (local address 127.29.106.62:42979)
0811 22:26:26.029739 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:26.029744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:26.029761 (+ 17us) server_negotiation.cc:407] Connection header received
0811 22:26:26.029818 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:26.029823 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:26.029882 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:26.029952 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:26.030510 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:26.031201 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:26.032374 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:26.032805 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:26.032889 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:26.036430 (+ 3541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.106.3:46805: BlockingRecv error: recv got EOF from 127.29.106.3:46805 (error 108)
Metrics: {"server-negotiator.queue_time_us":141}
W20260811 22:26:26.036691 15169 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 11 22:26:26 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.1@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:26 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.1@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:26:26.289088 15181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:26.280189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.1:51033)
0811 22:26:26.280728 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:26.280745 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:26.280878 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:26.281183 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:26.281186 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:26.281584 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:26.281734 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:26.281745 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:26.282935 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:26.282939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:26.284081 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:26.284095 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:26.284786 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:26.284790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:26.284901 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:26.285436 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:26.285452 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:26.288832 (+ 3380us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":295,"threads_started":1}
W20260811 22:26:26.289227 15177 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:26.280254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:51033 (local address 127.29.106.62:42979)
0811 22:26:26.280417 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:26.280421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:26.280820 (+ 399us) server_negotiation.cc:407] Connection header received
0811 22:26:26.280984 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:26.280988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:26.281070 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:26.281153 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:26.281866 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:26.282793 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:26.284227 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:26.284648 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:26.284744 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:26.289025 (+ 4281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.106.1:51033: BlockingRecv error: recv got EOF from 127.29.106.1:51033 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 11 22:26:26 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.2@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:26 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.2@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:26:26.634220 15182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:26.625698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.2:34333)
0811 22:26:26.626141 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:26.626157 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:26.626268 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:26.626524 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:26.626528 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:26.626887 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:26.627053 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:26.627063 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:26.628059 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:26.628063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:26.629183 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:26.629197 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:26.630117 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:26.630123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:26.630278 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:26.630836 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:26.630853 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:26.633993 (+ 3140us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":244,"threads_started":1}
W20260811 22:26:26.634378 15177 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:26.625774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:34333 (local address 127.29.106.62:42979)
0811 22:26:26.625994 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:26.626001 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:26.626226 (+ 225us) server_negotiation.cc:407] Connection header received
0811 22:26:26.626351 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:26.626354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:26.626416 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:26.626494 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:26.627172 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:26.627950 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:26.629320 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:26.629956 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:26.630069 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:26.634172 (+ 4103us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.106.2:34333: BlockingRecv error: recv got EOF from 127.29.106.2:34333 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 11 22:26:27 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.3@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:27 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.3@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:26:27.046391 15185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:27.037226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.3:36069)
0811 22:26:27.037735 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:27.037751 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:27.037879 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:27.038241 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:27.038246 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:27.038668 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:27.038820 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:27.038829 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:27.039883 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:27.039888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:27.040885 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:27.040898 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:27.041638 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:27.041644 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:27.041823 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:27.042388 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:27.042405 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:27.046063 (+ 3658us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":289,"threads_started":1}
W20260811 22:26:27.046512 15177 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:27.037317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:36069 (local address 127.29.106.62:42979)
0811 22:26:27.037524 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:27.037531 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:27.037834 (+ 303us) server_negotiation.cc:407] Connection header received
0811 22:26:27.038020 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:27.038025 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:27.038109 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:27.038191 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:27.038966 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:27.039757 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:27.041022 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:27.041461 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:27.041581 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:27.046281 (+ 4700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.106.3:36069: BlockingRecv error: recv got EOF from 127.29.106.3:36069 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 11 22:26:27 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.1@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:27 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.1@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:26:27.298038 15187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:27.289930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.1:46035)
0811 22:26:27.290377 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:27.290393 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:27.290516 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:27.290781 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:27.290784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:27.291210 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:27.291440 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:27.291451 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:27.292403 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:27.292407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:27.293317 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:27.293328 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:27.293931 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:27.293935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:27.294046 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:27.294565 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:27.294581 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:27.297804 (+ 3223us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":247,"threads_started":1}
W20260811 22:26:27.298242 15177 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:27.289999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:46035 (local address 127.29.106.62:42979)
0811 22:26:27.290219 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:27.290226 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:27.290461 (+ 235us) server_negotiation.cc:407] Connection header received
0811 22:26:27.290608 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:27.290612 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:27.290674 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:27.290751 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:27.291564 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:27.292303 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:27.293432 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:27.293820 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:27.293911 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:27.298019 (+ 4108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.106.1:46035: BlockingRecv error: recv got EOF from 127.29.106.1:46035 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 11 22:26:27 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.2@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:27 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.2@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:26:27.643282 15188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:27.635088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.2:50373)
0811 22:26:27.635599 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:27.635615 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:27.635743 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:27.636058 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:27.636062 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:27.636439 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:27.636609 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:27.636618 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:27.637524 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:27.637529 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:27.638477 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:27.638489 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:27.639104 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:27.639109 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:27.639234 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:27.639786 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:27.639804 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:27.643009 (+ 3205us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":278,"threads_started":1}
W20260811 22:26:27.643383 15177 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:27.635134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:50373 (local address 127.29.106.62:42979)
0811 22:26:27.635363 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:27.635373 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:27.635677 (+ 304us) server_negotiation.cc:407] Connection header received
0811 22:26:27.635867 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:27.635873 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:27.635941 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:27.636018 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:27.636728 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:27.637414 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:27.638586 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:27.638975 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:27.639072 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:27.643191 (+ 4119us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.106.2:50373: BlockingRecv error: recv got EOF from 127.29.106.2:50373 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 11 22:26:28 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.3@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:28 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.3@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:26:28.056339 15195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:28.047203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.3:40793)
0811 22:26:28.047733 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:28.047750 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:28.047872 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:28.048202 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:28.048205 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:28.048605 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:28.048819 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:28.048829 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:28.049930 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:28.049934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:28.050841 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:28.050853 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:28.051546 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:28.051550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:28.051672 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:28.052219 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:28.052237 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:28.056016 (+ 3779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":317,"threads_started":1}
W20260811 22:26:28.056504 15177 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:28.047341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:40793 (local address 127.29.106.62:42979)
0811 22:26:28.047490 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:28.047494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:28.047819 (+ 325us) server_negotiation.cc:407] Connection header received
0811 22:26:28.048004 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:28.048008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:28.048096 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:28.048172 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:28.049005 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:28.049775 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:28.050991 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:28.051416 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:28.051509 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:28.056232 (+ 4723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.106.3:40793: BlockingRecv error: recv got EOF from 127.29.106.3:40793 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 11 22:26:28 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.1@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:28 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.1@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:26:28.307564 15197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:28.298922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.1:34459)
0811 22:26:28.299490 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:28.299506 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:28.299653 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:28.299973 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:28.299978 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:28.300363 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:28.300525 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:28.300535 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:28.301535 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:28.301541 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:28.302484 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:28.302497 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:28.303215 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:28.303220 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:28.303436 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:28.304007 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:28.304024 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:28.307312 (+ 3288us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":284,"threads_started":1}
W20260811 22:26:28.307717 15177 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:28.298990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:34459 (local address 127.29.106.62:42979)
0811 22:26:28.299180 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:28.299186 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:28.299585 (+ 399us) server_negotiation.cc:407] Connection header received
0811 22:26:28.299766 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:28.299772 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:28.299839 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:28.299919 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:28.300671 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:28.301387 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:28.302622 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:28.303077 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:28.303184 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:28.307498 (+ 4314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.106.1:34459: BlockingRecv error: recv got EOF from 127.29.106.1:34459 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 11 22:26:28 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.2@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:28 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.2@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:26:28.652648 15199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:28.644221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.2:44085)
0811 22:26:28.644735 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:28.644755 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:28.644870 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:28.645153 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:28.645156 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:28.645562 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:28.645746 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:28.645756 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:28.646877 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:28.646881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:28.647826 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:28.647840 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:28.648493 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:28.648497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:28.648646 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:28.649166 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:28.649184 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:28.652422 (+ 3238us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":277,"threads_started":1}
W20260811 22:26:28.652829 15177 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:28.644274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:44085 (local address 127.29.106.62:42979)
0811 22:26:28.644443 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:28.644450 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:28.644825 (+ 375us) server_negotiation.cc:407] Connection header received
0811 22:26:28.644971 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:28.644975 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:28.645036 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:28.645123 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:28.645873 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:28.646745 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:28.647944 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:28.648374 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:28.648464 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:28.652602 (+ 4138us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.106.2:44085: BlockingRecv error: recv got EOF from 127.29.106.2:44085 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 11 22:26:29 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.3@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:29 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.3@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:26:29.069502 15202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:29.057293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.3:36643)
0811 22:26:29.057735 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:29.057750 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:29.057864 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:29.058153 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:29.058156 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:29.058527 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:29.058698 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:29.058707 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:29.060043 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:29.060048 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:29.061078 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:29.061092 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:29.061745 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:29.061749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:29.061889 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:29.066067 (+ 4178us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:29.066084 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:29.069264 (+ 3180us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":236,"threads_started":1}
W20260811 22:26:29.069638 15177 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:29.057389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:36643 (local address 127.29.106.62:42979)
0811 22:26:29.057604 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:29.057611 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:29.057823 (+ 212us) server_negotiation.cc:407] Connection header received
0811 22:26:29.057953 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:29.057957 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:29.058031 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:29.058122 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:29.058803 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:29.059897 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:29.061208 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:29.061631 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:29.061723 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:29.069443 (+ 7720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.106.3:36643: BlockingRecv error: recv got EOF from 127.29.106.3:36643 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 11 22:26:29 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.1@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:29 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.1@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:26:29.317205 15203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:29.308448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.1:47821)
0811 22:26:29.308985 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:29.309000 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:29.309128 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:29.309452 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:29.309457 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:29.309870 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:29.310037 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:29.310049 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:29.311033 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:29.311039 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:29.312085 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:29.312099 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:29.312869 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:29.312875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:29.313038 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:29.313681 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:29.313699 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:29.316930 (+ 3231us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":267,"threads_started":1}
W20260811 22:26:29.317343 15177 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:29.308514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:47821 (local address 127.29.106.62:42979)
0811 22:26:29.308727 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:29.308734 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:29.309091 (+ 357us) server_negotiation.cc:407] Connection header received
0811 22:26:29.309245 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:29.309249 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:29.309312 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:29.309393 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:29.310176 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:29.310906 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:29.312227 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:29.312733 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:29.312842 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:29.317130 (+ 4288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.106.1:47821: BlockingRecv error: recv got EOF from 127.29.106.1:47821 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Aug 11 22:26:29 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.2@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:29 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.2@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:26:29.661988 15204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:29.653497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.2:32877)
0811 22:26:29.653994 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:29.654010 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:29.654121 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:29.654382 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:29.654384 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:29.654774 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:29.654935 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:29.654944 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:29.656001 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:29.656005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:29.657056 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:29.657069 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:29.657707 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:29.657711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:29.657823 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:29.658349 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:29.658366 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:29.661731 (+ 3365us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":269,"threads_started":1}
W20260811 22:26:29.662142 15177 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:29.653568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:32877 (local address 127.29.106.62:42979)
0811 22:26:29.653726 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:29.653731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:29.654073 (+ 342us) server_negotiation.cc:407] Connection header received
0811 22:26:29.654205 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:29.654209 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:29.654269 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:29.654346 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:29.655057 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:29.655865 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:29.657170 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:29.657599 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:29.657691 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:29.661942 (+ 4251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.106.2:32877: BlockingRecv error: recv got EOF from 127.29.106.2:32877 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 11 22:26:30 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.3@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:30 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.3@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:26:30.079077 15207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:30.070350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.3:39681)
0811 22:26:30.070886 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:30.070905 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:30.071019 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:30.071331 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:30.071334 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:30.071742 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:30.071912 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:30.071921 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:30.072862 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:30.072866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:30.073854 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:30.073867 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:30.074531 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:30.074537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:30.074676 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:30.075303 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:30.075326 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:30.078822 (+ 3496us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":292,"threads_started":1}
W20260811 22:26:30.079293 15177 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:30.070447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:39681 (local address 127.29.106.62:42979)
0811 22:26:30.070604 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:30.070608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:30.070968 (+ 360us) server_negotiation.cc:407] Connection header received
0811 22:26:30.071109 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:30.071113 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:30.071196 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:30.071292 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:30.072021 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:30.072738 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:30.073963 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:30.074392 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:30.074477 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:30.079009 (+ 4532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.106.3:39681: BlockingRecv error: recv got EOF from 127.29.106.3:39681 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 11 22:26:30 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.1@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:30 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.1@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:26:30.326845 15208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:30.318067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.1:49669)
0811 22:26:30.318755 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:30.318776 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:30.318925 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:30.319330 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:30.319335 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:30.319901 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:30.320078 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:30.320088 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:30.321098 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:30.321102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:30.322108 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:30.322122 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:30.322750 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:30.322754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:30.322877 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:30.323477 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:30.323494 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:30.326617 (+ 3123us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":440,"threads_started":1}
W20260811 22:26:30.327003 15177 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:30.318166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:49669 (local address 127.29.106.62:42979)
0811 22:26:30.318369 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:30.318377 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:30.318856 (+ 479us) server_negotiation.cc:407] Connection header received
0811 22:26:30.319075 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:30.319079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:30.319161 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:30.319299 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:30.320205 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:30.320969 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:30.322228 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:30.322642 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:30.322731 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:30.326809 (+ 4078us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.106.1:49669: BlockingRecv error: recv got EOF from 127.29.106.1:49669 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 11 22:26:30 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.2@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:30 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.2@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:26:30.671413 15209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:30.662817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.2:40837)
0811 22:26:30.663328 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:30.663345 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:30.663492 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:30.663836 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:30.663841 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:30.664270 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:30.664423 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:30.664433 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:30.665448 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:30.665454 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:30.666413 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:30.666426 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:30.667175 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:30.667181 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:30.667400 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:30.667921 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:30.667937 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:30.671143 (+ 3206us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":247,"threads_started":1}
W20260811 22:26:30.671599 15177 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:30.662988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:40837 (local address 127.29.106.62:42979)
0811 22:26:30.663157 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:30.663162 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:30.663425 (+ 263us) server_negotiation.cc:407] Connection header received
0811 22:26:30.663625 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:30.663631 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:30.663699 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:30.663781 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:30.664564 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:30.665312 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:30.666545 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:30.667019 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:30.667134 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:30.671367 (+ 4233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.106.2:40837: BlockingRecv error: recv got EOF from 127.29.106.2:40837 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 11 22:26:31 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.3@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:31 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.3@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:26:31.089205 15210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:31.080040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.3:54057)
0811 22:26:31.080615 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:31.080632 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:31.080794 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:31.081134 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:31.081137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:31.081605 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:31.081755 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:31.081764 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:31.082973 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:31.082978 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:31.084161 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:31.084175 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:31.084879 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:31.084884 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:31.085064 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:31.085643 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:31.085661 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:31.088902 (+ 3241us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":367,"threads_started":1}
W20260811 22:26:31.089358 15177 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:31.080157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:54057 (local address 127.29.106.62:42979)
0811 22:26:31.080474 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:31.080481 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:31.080741 (+ 260us) server_negotiation.cc:407] Connection header received
0811 22:26:31.080897 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:31.080900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:31.081003 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:31.081095 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:31.081990 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:31.082807 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:31.084329 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:31.084739 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:31.084862 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:31.089129 (+ 4267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.106.3:54057: BlockingRecv error: recv got EOF from 127.29.106.3:54057 (error 108)
Metrics: {"server-negotiator.queue_time_us":168}
Aug 11 22:26:31 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.1@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:31 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.1@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:26:31.337082 15211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:31.327834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.1:58909)
0811 22:26:31.328331 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:31.328347 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:31.328468 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:31.328806 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:31.328811 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:31.329369 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:31.329615 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:31.329627 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:31.330746 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:31.330750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:31.331726 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:31.331739 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:31.332435 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:31.332439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:31.332568 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:31.333143 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:31.333159 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:31.336735 (+ 3576us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":292,"threads_started":1}
W20260811 22:26:31.337208 15177 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:31.327935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:58909 (local address 127.29.106.62:42979)
0811 22:26:31.328195 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:31.328202 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:31.328453 (+ 251us) server_negotiation.cc:407] Connection header received
0811 22:26:31.328571 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:31.328574 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:31.328640 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:31.328713 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:31.329736 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:31.330622 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:31.331849 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:31.332288 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:31.332391 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:31.336962 (+ 4571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.106.1:58909: BlockingRecv error: recv got EOF from 127.29.106.1:58909 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Aug 11 22:26:31 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.2@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:31 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.2@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:26:31.680974 15212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:31.672280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.2:48611)
0811 22:26:31.672704 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:31.672720 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:31.672833 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:31.673114 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:31.673117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:31.673519 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:31.673684 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:31.673693 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:31.674603 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:31.674607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:31.675567 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:31.675579 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:31.676256 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:31.676263 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:31.676445 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:31.677073 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:31.677099 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:31.680629 (+ 3530us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":218,"threads_started":1}
W20260811 22:26:31.681037 15177 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:31.672328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:48611 (local address 127.29.106.62:42979)
0811 22:26:31.672565 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:31.672572 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:31.672793 (+ 221us) server_negotiation.cc:407] Connection header received
0811 22:26:31.672922 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:31.672926 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:31.673001 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:31.673083 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:31.673800 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:31.674476 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:31.675672 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:31.676102 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:31.676231 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:31.680833 (+ 4602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.106.2:48611: BlockingRecv error: recv got EOF from 127.29.106.2:48611 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 11 22:26:32 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.3@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:32 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.3@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:26:32.098701 15213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:32.090196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.3:52153)
0811 22:26:32.090629 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:32.090645 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:32.090757 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:32.091048 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:32.091051 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:32.091570 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:32.091762 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:32.091775 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:32.092980 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:32.092984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:32.093909 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:32.093922 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:32.094514 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:32.094517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:32.094648 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:32.095197 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:32.095214 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:32.098467 (+ 3253us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":221,"threads_started":1}
W20260811 22:26:32.098924 15177 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:32.090274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:52153 (local address 127.29.106.62:42979)
0811 22:26:32.090473 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:32.090479 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:32.090717 (+ 238us) server_negotiation.cc:407] Connection header received
0811 22:26:32.090844 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:32.090847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:32.090923 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:32.091002 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:32.091907 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:32.092855 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:32.094015 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:32.094404 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:32.094490 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:32.098689 (+ 4199us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.106.3:52153: BlockingRecv error: recv got EOF from 127.29.106.3:52153 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 11 22:26:32 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.1@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:32 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.1@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:26:32.346804 15215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:32.337923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.1:41431)
0811 22:26:32.338459 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:32.338475 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:32.338607 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:32.338926 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:32.338930 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:32.339370 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:32.339545 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:32.339555 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:32.340612 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:32.340616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:32.341598 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:32.341611 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:32.342387 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:32.342393 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:32.342561 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:32.343076 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:32.343092 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:32.346482 (+ 3390us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":240,"threads_started":1}
W20260811 22:26:32.346928 15177 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:32.338008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:41431 (local address 127.29.106.62:42979)
0811 22:26:32.338178 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:32.338184 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:32.338560 (+ 376us) server_negotiation.cc:407] Connection header received
0811 22:26:32.338717 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:32.338723 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:32.338787 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:32.338865 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:32.339684 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:32.340479 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:32.341732 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:32.342246 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:32.342353 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:32.346685 (+ 4332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.106.1:41431: BlockingRecv error: recv got EOF from 127.29.106.1:41431 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 11 22:26:32 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.2@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:32 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.2@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:26:32.690954 15216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:32.682051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.2:42469)
0811 22:26:32.682618 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:32.682635 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:32.682780 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:32.683173 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:32.683177 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:32.683752 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:32.683971 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:32.683986 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:32.685042 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:32.685046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:32.685990 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:32.686002 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:32.686663 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:32.686667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:32.686776 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:32.687380 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:32.687397 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:32.690688 (+ 3291us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":257,"threads_started":1}
W20260811 22:26:32.691118 15177 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:32.682126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:42469 (local address 127.29.106.62:42979)
0811 22:26:32.682327 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:32.682334 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:32.682727 (+ 393us) server_negotiation.cc:407] Connection header received
0811 22:26:32.682918 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:32.682923 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:32.683010 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:32.683110 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:32.684131 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:32.684919 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:32.686110 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:32.686549 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:32.686641 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:32.690898 (+ 4257us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.106.2:42469: BlockingRecv error: recv got EOF from 127.29.106.2:42469 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 11 22:26:33 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.3@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:33 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.3@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:26:33.108137 15218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:33.099606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.3:48289)
0811 22:26:33.100151 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:33.100167 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:33.100282 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:33.100582 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:33.100585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:33.101004 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:33.101191 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:33.101201 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:33.102158 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:33.102162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:33.103323 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:33.103336 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:33.104025 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:33.104031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:33.104178 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:33.104665 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:33.104681 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:33.107896 (+ 3215us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":257,"threads_started":1}
W20260811 22:26:33.108311 15177 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:33.099664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:48289 (local address 127.29.106.62:42979)
0811 22:26:33.099855 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:33.099862 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:33.100254 (+ 392us) server_negotiation.cc:407] Connection header received
0811 22:26:33.100381 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:33.100385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:33.100473 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:33.100558 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:33.101300 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:33.102031 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:33.103457 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:33.103881 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:33.103992 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:33.108103 (+ 4111us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.106.3:48289: BlockingRecv error: recv got EOF from 127.29.106.3:48289 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 11 22:26:33 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.1@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:33 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.1@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:26:33.356421 15219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:33.348040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.1:58939)
0811 22:26:33.348608 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:33.348623 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:33.348739 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:33.349003 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:33.349006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:33.349426 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:33.349602 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:33.349612 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:33.350612 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:33.350615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:33.351586 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:33.351600 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:33.352257 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:33.352261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:33.352375 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:33.352906 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:33.352923 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:33.356180 (+ 3257us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":287,"threads_started":1}
W20260811 22:26:33.356554 15177 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:33.348214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:58939 (local address 127.29.106.62:42979)
0811 22:26:33.348400 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:33.348404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:33.348687 (+ 283us) server_negotiation.cc:407] Connection header received
0811 22:26:33.348826 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:33.348835 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:33.348897 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:33.348975 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:33.349718 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:33.350497 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:33.351698 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:33.352118 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:33.352207 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:33.356347 (+ 4140us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.106.1:58939: BlockingRecv error: recv got EOF from 127.29.106.1:58939 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 11 22:26:33 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.2@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:33 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.2@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:26:33.700657 15221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:33.691812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.2:33957)
0811 22:26:33.692419 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:33.692434 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:33.692561 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:33.692854 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:33.692856 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:33.693274 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:33.693455 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:33.693464 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:33.695077 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:33.695082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:33.696057 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:33.696071 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:33.696701 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:33.696705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:33.696825 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:33.697338 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:33.697355 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:33.700403 (+ 3048us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":341,"threads_started":1}
W20260811 22:26:33.700855 15177 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:33.691925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:33957 (local address 127.29.106.62:42979)
0811 22:26:33.692080 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:33.692085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:33.692498 (+ 413us) server_negotiation.cc:407] Connection header received
0811 22:26:33.692648 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:33.692651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:33.692746 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:33.692826 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:33.693590 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:33.694916 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:33.696173 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:33.696569 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:33.696658 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:33.700629 (+ 3971us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.106.2:33957: BlockingRecv error: recv got EOF from 127.29.106.2:33957 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 11 22:26:34 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.3@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:34 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.3@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:26:34.122239 15223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:34.108934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.3:43681)
0811 22:26:34.109523 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:34.109543 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:34.109671 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:34.110025 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:34.110031 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:34.110541 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:34.110760 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:34.110770 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:34.111757 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:34.111761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:34.112696 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:34.112709 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:34.113397 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:34.113401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:34.113509 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:34.118427 (+ 4918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:34.118445 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:34.121884 (+ 3439us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":296,"threads_started":1}
W20260811 22:26:34.122542 15177 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:34.109031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:43681 (local address 127.29.106.62:42979)
0811 22:26:34.109288 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:34.109293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:34.109621 (+ 328us) server_negotiation.cc:407] Connection header received
0811 22:26:34.109788 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:34.109794 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:34.109868 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:34.109946 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:34.110886 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:34.111653 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:34.112819 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:34.113263 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:34.113352 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:34.122119 (+ 8767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.106.3:43681: BlockingRecv error: recv got EOF from 127.29.106.3:43681 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Aug 11 22:26:34 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.1@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:34 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.1@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:26:34.365842 15225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:34.357293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.1:54037)
0811 22:26:34.357771 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:34.357787 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:34.357901 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:34.358161 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:34.358164 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:34.358583 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:34.358733 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:34.358743 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:34.359725 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:34.359729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:34.360677 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:34.360689 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:34.361300 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:34.361304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:34.361425 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:34.361940 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:34.361955 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:34.365545 (+ 3590us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":242,"threads_started":1}
W20260811 22:26:34.365957 15177 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:34.357344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:54037 (local address 127.29.106.62:42979)
0811 22:26:34.357556 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:34.357562 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:34.357859 (+ 297us) server_negotiation.cc:407] Connection header received
0811 22:26:34.357987 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:34.357991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:34.358058 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:34.358133 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:34.358841 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:34.359620 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:34.360780 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:34.361193 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:34.361278 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:34.365749 (+ 4471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.106.1:54037: BlockingRecv error: recv got EOF from 127.29.106.1:54037 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 11 22:26:34 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.2@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:34 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.2@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:26:34.710211 15229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:34.701504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.2:53863)
0811 22:26:34.702014 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:34.702029 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:34.702163 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:34.702483 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:34.702487 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:34.702914 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:34.703119 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:34.703132 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:34.704216 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:34.704220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:34.705163 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:34.705175 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:34.705778 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:34.705782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:34.705920 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:34.706484 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:34.706501 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:34.709983 (+ 3482us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":273,"threads_started":1}
W20260811 22:26:34.710376 15177 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:34.701583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:53863 (local address 127.29.106.62:42979)
0811 22:26:34.701759 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:34.701766 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:34.702116 (+ 350us) server_negotiation.cc:407] Connection header received
0811 22:26:34.702280 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:34.702286 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:34.702352 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:34.702429 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:34.703303 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:34.704088 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:34.705264 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:34.705659 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:34.705744 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:34.710160 (+ 4416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.106.2:53863: BlockingRecv error: recv got EOF from 127.29.106.2:53863 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 11 22:26:35 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.3@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:35 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.3@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:26:35.131547 15232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:35.123119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.3:47383)
0811 22:26:35.123649 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:35.123664 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:35.123795 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:35.124081 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:35.124084 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:35.124496 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:35.124646 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:35.124656 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:35.125634 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:35.125639 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:35.126598 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:35.126612 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:35.127329 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:35.127335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:35.127482 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:35.128015 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:35.128032 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:35.131326 (+ 3294us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":290,"threads_started":1}
W20260811 22:26:35.131722 15177 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:35.123173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:47383 (local address 127.29.106.62:42979)
0811 22:26:35.123408 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:35.123413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:35.123729 (+ 316us) server_negotiation.cc:407] Connection header received
0811 22:26:35.123898 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:35.123902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:35.123965 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:35.124043 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:35.124778 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:35.125481 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:35.126734 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:35.127189 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:35.127329 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:35.131518 (+ 4189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.106.3:47383: BlockingRecv error: recv got EOF from 127.29.106.3:47383 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 11 22:26:35 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.1@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:35 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.1@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:26:35.384352 15233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:35.366785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.1:42007)
0811 22:26:35.374575 (+ 7790us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:35.374597 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:35.374748 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:35.375175 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:35.375180 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:35.375749 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:35.375925 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:35.375936 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:35.376940 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:35.376946 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:35.378078 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:35.378096 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:35.379151 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:35.379157 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:35.379352 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:35.379982 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:35.380004 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:35.383992 (+ 3988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":7651,"thread_start_us":7508,"threads_started":1}
W20260811 22:26:35.384423 15177 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:35.366869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:42007 (local address 127.29.106.62:42979)
0811 22:26:35.367111 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:35.367117 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:35.374686 (+ 7569us) server_negotiation.cc:407] Connection header received
0811 22:26:35.374869 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:35.374874 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:35.375004 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:35.375102 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:35.376064 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:35.376801 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:35.378211 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:35.378901 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:35.379038 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:35.384193 (+ 5155us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.106.1:42007: BlockingRecv error: recv got EOF from 127.29.106.1:42007 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Aug 11 22:26:35 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.2@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:35 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.2@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:26:35.719501 15234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:35.711002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.2:43935)
0811 22:26:35.711481 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:35.711498 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:35.711618 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:35.711877 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:35.711880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:35.712251 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:35.712419 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:35.712428 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:35.713337 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:35.713340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:35.714329 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:35.714341 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:35.715181 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:35.715187 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:35.715394 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:35.715951 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:35.715970 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:35.719156 (+ 3186us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":260,"threads_started":1}
W20260811 22:26:35.719632 15177 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:35.711091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:43935 (local address 127.29.106.62:42979)
0811 22:26:35.711323 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:35.711329 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:35.711573 (+ 244us) server_negotiation.cc:407] Connection header received
0811 22:26:35.711708 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:35.711711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:35.711774 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:35.711850 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:35.712526 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:35.713229 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:35.714456 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:35.715024 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:35.715137 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:35.719396 (+ 4259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.106.2:43935: BlockingRecv error: recv got EOF from 127.29.106.2:43935 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 11 22:26:36 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.3@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:36 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.3@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:26:36.140985 15236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:36.132357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.3:46961)
0811 22:26:36.132859 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:36.132875 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:36.132982 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:36.133246 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:36.133249 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:36.133626 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:36.133799 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:36.133807 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:36.134792 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:36.134796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:36.135823 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:36.135835 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:36.136658 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:36.136662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:36.136783 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:36.137402 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:36.137421 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:36.140731 (+ 3310us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":287,"threads_started":1}
W20260811 22:26:36.141089 15177 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:36.132497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:46961 (local address 127.29.106.62:42979)
0811 22:26:36.132720 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:36.132726 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:36.132943 (+ 217us) server_negotiation.cc:407] Connection header received
0811 22:26:36.133072 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:36.133076 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:36.133136 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:36.133214 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:36.133902 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:36.134672 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:36.135947 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:36.136385 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:36.136623 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:36.140887 (+ 4264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.106.3:46961: BlockingRecv error: recv got EOF from 127.29.106.3:46961 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Aug 11 22:26:36 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.1@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:36 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.1@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:26:36.393879 15241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:36.385400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.1:54919)
0811 22:26:36.385889 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:36.385905 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:36.386045 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:36.386379 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:36.386384 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:36.386801 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:36.386987 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:36.386997 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:36.388052 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:36.388056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:36.388961 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:36.388972 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:36.389674 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:36.389680 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:36.389844 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:36.390380 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:36.390396 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:36.393575 (+ 3179us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":258,"threads_started":1}
W20260811 22:26:36.394013 15177 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:36.385474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:54919 (local address 127.29.106.62:42979)
0811 22:26:36.385684 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:36.385691 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:36.385993 (+ 302us) server_negotiation.cc:407] Connection header received
0811 22:26:36.386159 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:36.386165 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:36.386236 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:36.386328 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:36.387125 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:36.387881 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:36.389092 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:36.389533 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:36.389638 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:36.393785 (+ 4147us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.106.1:54919: BlockingRecv error: recv got EOF from 127.29.106.1:54919 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 11 22:26:36 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.2@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:36 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.2@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:26:36.729166 15242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:36.720400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.2:59181)
0811 22:26:36.720907 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:36.720923 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:36.721160 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:36.721500 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:36.721504 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:36.722091 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:36.722326 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:36.722338 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:36.723400 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:36.723404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:36.724318 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:36.724330 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:36.724944 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:36.724948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:36.725063 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:36.725585 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:36.725601 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:36.728939 (+ 3338us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":274,"threads_started":1}
W20260811 22:26:36.729295 15177 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:36.720501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:59181 (local address 127.29.106.62:42979)
0811 22:26:36.720704 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:36.720710 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:36.721044 (+ 334us) server_negotiation.cc:407] Connection header received
0811 22:26:36.721288 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:36.721292 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:36.721355 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:36.721443 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:36.722466 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:36.723241 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:36.724430 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:36.724830 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:36.724916 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:36.729091 (+ 4175us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.106.2:59181: BlockingRecv error: recv got EOF from 127.29.106.2:59181 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 11 22:26:37 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.3@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:37 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.3@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:26:37.150651 15243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:37.141854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.3:60337)
0811 22:26:37.142350 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:37.142365 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:37.142495 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:37.142816 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:37.142821 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:37.143291 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:37.143475 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:37.143485 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:37.144470 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:37.144474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:37.145412 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:37.145425 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:37.146178 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:37.146183 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:37.146345 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:37.146935 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:37.146951 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:37.150442 (+ 3491us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":278,"threads_started":1}
W20260811 22:26:37.150799 15177 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:37.141942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:60337 (local address 127.29.106.62:42979)
0811 22:26:37.142142 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:37.142148 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:37.142448 (+ 300us) server_negotiation.cc:407] Connection header received
0811 22:26:37.142615 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:37.142621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:37.142686 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:37.142766 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:37.143616 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:37.144343 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:37.145559 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:37.146028 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:37.146132 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:37.150603 (+ 4471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.106.3:60337: BlockingRecv error: recv got EOF from 127.29.106.3:60337 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 11 22:26:37 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.1@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:37 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.1@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:26:37.403931 15244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:37.394895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.1:51621)
0811 22:26:37.395470 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:37.395487 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:37.395603 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:37.395879 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:37.395882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:37.396294 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:37.396469 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:37.396478 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:37.397484 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:37.397487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:37.398584 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:37.398599 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:37.399293 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:37.399300 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:37.399441 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:37.399995 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:37.400011 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:37.403518 (+ 3507us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":289,"threads_started":1}
W20260811 22:26:37.403961 15177 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:37.394953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:51621 (local address 127.29.106.62:42979)
0811 22:26:37.395203 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:37.395209 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:37.395560 (+ 351us) server_negotiation.cc:407] Connection header received
0811 22:26:37.395692 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:37.395696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:37.395772 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:37.395848 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:37.396593 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:37.397382 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:37.398697 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:37.399128 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:37.399220 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:37.403707 (+ 4487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.106.1:51621: BlockingRecv error: recv got EOF from 127.29.106.1:51621 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Aug 11 22:26:37 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.2@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:37 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.2@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:26:37.738937 15246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:37.730029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.2:48561)
0811 22:26:37.730505 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:37.730520 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:37.730635 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:37.730918 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:37.730922 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:37.731381 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:37.731565 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:37.731576 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:37.732633 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:37.732637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:37.733572 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:37.733590 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:37.734333 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:37.734338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:37.734468 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:37.735025 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:37.735043 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:37.738678 (+ 3635us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":229,"threads_started":1}
W20260811 22:26:37.739101 15177 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:37.730109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:48561 (local address 127.29.106.62:42979)
0811 22:26:37.730332 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:37.730338 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:37.730597 (+ 259us) server_negotiation.cc:407] Connection header received
0811 22:26:37.730725 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:37.730729 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:37.730792 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:37.730868 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:37.731704 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:37.732507 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:37.733729 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:37.734163 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:37.734263 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:37.738849 (+ 4586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.106.2:48561: BlockingRecv error: recv got EOF from 127.29.106.2:48561 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Aug 11 22:26:38 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.3@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:38 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.3@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:26:38.160372 15248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:38.151534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.3:57681)
0811 22:26:38.152120 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:38.152136 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:38.152248 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:38.152510 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:38.152512 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:38.152938 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:38.153114 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:38.153123 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:38.154199 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:38.154203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:38.155406 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:38.155424 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:38.156178 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:38.156183 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:38.156314 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:38.156846 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:38.156863 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:38.160128 (+ 3265us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":319,"threads_started":1}
W20260811 22:26:38.160553 15177 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:38.151624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:57681 (local address 127.29.106.62:42979)
0811 22:26:38.151782 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:38.151787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:38.152194 (+ 407us) server_negotiation.cc:407] Connection header received
0811 22:26:38.152347 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:38.152350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:38.152411 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:38.152481 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:38.153226 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:38.154063 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:38.155551 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:38.156025 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:38.156126 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:38.160320 (+ 4194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.106.3:57681: BlockingRecv error: recv got EOF from 127.29.106.3:57681 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 11 22:26:38 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.1@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:38 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.1@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:26:38.413517 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:38.404821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.1:43349)
0811 22:26:38.405297 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:38.405312 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:38.405450 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:38.405759 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:38.405766 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:38.406182 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:38.406337 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:38.406347 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:38.407369 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:38.407373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:38.408386 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:38.408403 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:38.409144 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:38.409150 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:38.409333 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:38.409920 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:38.409938 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:38.413210 (+ 3272us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":239,"threads_started":1}
W20260811 22:26:38.413666 15177 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:38.404917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:43349 (local address 127.29.106.62:42979)
0811 22:26:38.405156 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:38.405163 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:38.405395 (+ 232us) server_negotiation.cc:407] Connection header received
0811 22:26:38.405564 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:38.405569 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:38.405629 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:38.405708 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:38.406488 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:38.407216 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:38.408528 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:38.409004 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:38.409139 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:38.413445 (+ 4306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.106.1:43349: BlockingRecv error: recv got EOF from 127.29.106.1:43349 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 11 22:26:38 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.2@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:38 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.2@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:26:38.748642 15253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:38.739797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.2:45301)
0811 22:26:38.740314 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:38.740330 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:38.740460 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:38.740792 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:38.740798 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:38.741260 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:38.741432 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:38.741444 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:38.742649 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:38.742655 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:38.743638 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:38.743652 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:38.744332 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:38.744338 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:38.744507 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:38.745101 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:38.745119 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:38.748336 (+ 3217us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":269,"threads_started":1}
W20260811 22:26:38.748761 15177 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:38.739937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:45301 (local address 127.29.106.62:42979)
0811 22:26:38.740156 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:38.740162 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:38.740401 (+ 239us) server_negotiation.cc:407] Connection header received
0811 22:26:38.740590 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:38.740594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:38.740656 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:38.740738 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:38.741578 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:38.742505 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:38.743769 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:38.744197 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:38.744304 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:38.748535 (+ 4231us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.106.2:45301: BlockingRecv error: recv got EOF from 127.29.106.2:45301 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 11 22:26:39 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.3@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:39 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.3@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:26:39.173678 15254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:39.161173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.3:33019)
0811 22:26:39.161772 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:39.161788 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:39.161918 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:39.162195 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:39.162197 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:39.162608 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:39.162760 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:39.162771 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:39.163772 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:39.163776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:39.164697 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:39.164709 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:39.165319 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:39.165323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:39.165441 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:39.169786 (+ 4345us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:39.169803 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:39.173441 (+ 3638us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":393,"threads_started":1}
W20260811 22:26:39.173833 15177 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:39.161241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:33019 (local address 127.29.106.62:42979)
0811 22:26:39.161439 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:39.161447 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:39.161896 (+ 449us) server_negotiation.cc:407] Connection header received
0811 22:26:39.162013 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:39.162017 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:39.162081 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:39.162164 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:39.162890 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:39.163665 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:39.164817 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:39.165215 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:39.165306 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:39.173637 (+ 8331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.106.3:33019: BlockingRecv error: recv got EOF from 127.29.106.3:33019 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 11 22:26:39 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.1@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:39 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.1@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:26:39.423547 15256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:39.414753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.1:56669)
0811 22:26:39.415318 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:39.415340 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:39.415481 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:39.415824 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:39.415828 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:39.416314 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:39.416520 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:39.416530 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:39.417486 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:39.417490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:39.418420 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:39.418432 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:39.419108 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:39.419112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:39.419240 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:39.419841 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:39.419859 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:39.423190 (+ 3331us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":289,"threads_started":1}
W20260811 22:26:39.423705 15177 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:39.414804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:56669 (local address 127.29.106.62:42979)
0811 22:26:39.415189 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:39.415195 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:39.415423 (+ 228us) server_negotiation.cc:407] Connection header received
0811 22:26:39.415599 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:39.415604 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:39.415683 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:39.415764 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:39.416657 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:39.417371 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:39.418530 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:39.418984 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:39.419077 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:39.423431 (+ 4354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.106.1:56669: BlockingRecv error: recv got EOF from 127.29.106.1:56669 (error 108)
Metrics: {"server-negotiator.queue_time_us":221}
Aug 11 22:26:39 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.2@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:26:39 dist-test-slave-gjrz krb5kdc[13378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.106.2@KRBTEST.COM for oryx/127.29.106.62@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:26:39.758492 15258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:39.749874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42979 (local address 127.29.106.2:33639)
0811 22:26:39.750393 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:39.750409 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:39.750519 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:39.750853 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:39.750858 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:39.751238 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:39.751466 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:39.751477 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:39.752450 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:39.752454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:39.753428 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:39.753442 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:39.754123 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:39.754129 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:39.754299 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:39.754787 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:39.754804 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:39.758168 (+ 3364us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.106.62:42979: Server oryx/127.29.106.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":278,"threads_started":1}
W20260811 22:26:39.758610 15177 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:39.749959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:33639 (local address 127.29.106.62:42979)
0811 22:26:39.750150 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:39.750156 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:39.750484 (+ 328us) server_negotiation.cc:407] Connection header received
0811 22:26:39.750652 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:39.750656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:39.750717 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:39.750795 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:39.751608 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:39.752310 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:39.753562 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:39.753984 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:39.754102 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:39.758387 (+ 4285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.106.2:33639: BlockingRecv error: recv got EOF from 127.29.106.2:33639 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260811 22:26:39.934577 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 14404
I20260811 22:26:39.946774 15261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:39.935666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45272 (local address 127.29.106.1:44601)
0811 22:26:39.936262 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:39.936271 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:39.936298 (+ 27us) server_negotiation.cc:407] Connection header received
0811 22:26:39.936362 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:39.936368 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:39.936611 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:39.936775 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:39.938115 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:39.939202 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:39.940415 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:39.941048 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:39.941144 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:39.943542 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:39.943550 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:39.943564 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:39.943603 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:39.946239 (+ 2636us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.29.106.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0811 22:26:39.946558 (+ 319us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45272: Request ticket server kudu/127.29.106.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":289,"threads_started":1}
W20260811 22:26:39.946879 15261 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45272: Request ticket server kudu/127.29.106.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260811 22:26:39.946926 30120 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 14404: Not authorized: Client connection negotiation failed: client connection to 127.29.106.1:44601: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.29.106.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260811 22:26:39.947118 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 14404
I20260811 22:26:39.975545 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 14682
I20260811 22:26:39.988531 15263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:39.976701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56066 (local address 127.29.106.2:36315)
0811 22:26:39.977446 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:39.977455 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:39.977490 (+ 35us) server_negotiation.cc:407] Connection header received
0811 22:26:39.977571 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:39.977580 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:39.977907 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:39.978135 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:39.979434 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:39.980949 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:39.982290 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:39.982851 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:39.982951 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:39.985311 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:39.985319 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:39.985333 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:39.985375 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:39.987995 (+ 2620us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.29.106.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0811 22:26:39.988283 (+ 288us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56066: Request ticket server kudu/127.29.106.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":349,"threads_started":1}
W20260811 22:26:39.988596 30120 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 14682: Not authorized: Client connection negotiation failed: client connection to 127.29.106.2:36315: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.29.106.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260811 22:26:39.988638 15263 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56066: Request ticket server kudu/127.29.106.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260811 22:26:39.988770 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 14682
I20260811 22:26:40.010557 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 14933
I20260811 22:26:40.022969 15264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:40.011643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51798 (local address 127.29.106.3:43935)
0811 22:26:40.012352 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:40.012362 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:40.012393 (+ 31us) server_negotiation.cc:407] Connection header received
0811 22:26:40.012467 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:40.012475 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:40.012806 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:40.013014 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:40.014276 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:40.015508 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:40.016670 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:40.017203 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:40.017325 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:40.019723 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:40.019733 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:40.019747 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:40.019798 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:40.022428 (+ 2630us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.29.106.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0811 22:26:40.022713 (+ 285us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51798: Request ticket server kudu/127.29.106.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":331,"threads_started":1}
W20260811 22:26:40.023080 15264 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51798: Request ticket server kudu/127.29.106.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260811 22:26:40.023140 30120 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 14933: Not authorized: Client connection negotiation failed: client connection to 127.29.106.3:43935: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.29.106.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260811 22:26:40.023453 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 14933
I20260811 22:26:40.053953 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 14286
I20260811 22:26:40.055361 14393 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:26:40.178270 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 14286
2026-08-11T22:26:40Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21282 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-11T22:26:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:26:40Z Disabled control of system clock
I20260811 22:26:40.267205 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:41115
--webserver_interface=127.29.106.62
--webserver_port=0
--builtin_ntp_servers=127.29.106.20:41783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:41115
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260811 22:26:40.591199 15269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:26:40.591467 15269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:26:40.591557 15269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:26:40.599525 15269 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:26:40.599627 15269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:26:40.599668 15269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:26:40.599723 15269 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:26:40.599773 15269 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:26:40.608608 15269 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:41783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:41115
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:41115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.106.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:26:40.609732 15269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:26:40.611400 15269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:26:40.622493 15281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:40.622521 15280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:40.622439 15287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:26:40.623042 15269 server_base.cc:1034] running on GCE node
I20260811 22:26:40.623834 15269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:26:40.625092 15269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:26:40.626278 15269 hybrid_clock.cc:648] HybridClock initialized: now 1786487200626231 us; error 44 us; skew 500 ppm
I20260811 22:26:40.626610 15269 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 22:26:40.628739 15269 webserver.cc:460] Webserver started at http://127.29.106.62:35185/ using document root <none> and password file <none>
I20260811 22:26:40.629385 15269 fs_manager.cc:359] Metadata directory not provided
I20260811 22:26:40.629480 15269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:26:40.629808 15269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:26:40.632680 15269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "941b88a30258457b9ab7c2d3101ca378"
format_stamp: "Formatted at 2026-08-11 22:26:40 on dist-test-slave-gjrz"
server_key: "febc9ed613f02f0af94cd2d68ddf67aa"
server_key_iv: "64bedbf36e3af87fe1452aef744f3a29"
server_key_version: "encryptionkey@0"
I20260811 22:26:40.633435 15269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "941b88a30258457b9ab7c2d3101ca378"
format_stamp: "Formatted at 2026-08-11 22:26:40 on dist-test-slave-gjrz"
server_key: "febc9ed613f02f0af94cd2d68ddf67aa"
server_key_iv: "64bedbf36e3af87fe1452aef744f3a29"
server_key_version: "encryptionkey@0"
I20260811 22:26:40.638185 15269 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260811 22:26:40.641405 15295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:40.642625 15269 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260811 22:26:40.642774 15269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "941b88a30258457b9ab7c2d3101ca378"
format_stamp: "Formatted at 2026-08-11 22:26:40 on dist-test-slave-gjrz"
server_key: "febc9ed613f02f0af94cd2d68ddf67aa"
server_key_iv: "64bedbf36e3af87fe1452aef744f3a29"
server_key_version: "encryptionkey@0"
I20260811 22:26:40.642961 15269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:26:40.661331 15269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:26:40.662061 15269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:26:40.662307 15269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:26:40.678743 15269 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.62:41115
I20260811 22:26:40.678776 15382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.62:41115 every 8 connection(s)
I20260811 22:26:40.679942 15269 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:26:40.683398 15384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:26:40.686563 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 15269
I20260811 22:26:40.686779 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:26:40.687626 30120 external_mini_cluster.cc:1428] Setting key d496b4fc39da0520d366f8fca7f54d80
I20260811 22:26:40.695844 15384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 941b88a30258457b9ab7c2d3101ca378: Bootstrap starting.
I20260811 22:26:40.699224 15384 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 941b88a30258457b9ab7c2d3101ca378: Neither blocks nor log segments found. Creating new log.
I20260811 22:26:40.700348 15384 log.cc:824] T 00000000000000000000000000000000 P 941b88a30258457b9ab7c2d3101ca378: Log is configured to *not* fsync() on all Append() calls
I20260811 22:26:40.700734 15389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:40.689602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56898 (local address 127.29.106.62:41115)
0811 22:26:40.690465 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:40.690475 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:40.690514 (+ 39us) server_negotiation.cc:407] Connection header received
0811 22:26:40.692156 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:40.692179 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:40.692635 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:40.692936 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:40.693596 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:40.694811 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:40.696642 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:40.697293 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:40.697420 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:40.698598 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:40.698605 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:40.698618 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:26:40.699100 (+ 482us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:40.699191 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:26:40.699214 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:40.699352 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:40.699534 (+ 182us) server_negotiation.cc:299] Negotiation successful
0811 22:26:40.699831 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":318,"threads_started":1}
I20260811 22:26:40.703114 15384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 941b88a30258457b9ab7c2d3101ca378: No bootstrap required, opened a new log
I20260811 22:26:40.709959 15384 raft_consensus.cc:348] T 00000000000000000000000000000000 P 941b88a30258457b9ab7c2d3101ca378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "941b88a30258457b9ab7c2d3101ca378" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 41115 } }
I20260811 22:26:40.710280 15384 raft_consensus.cc:374] T 00000000000000000000000000000000 P 941b88a30258457b9ab7c2d3101ca378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:26:40.710372 15384 raft_consensus.cc:729] T 00000000000000000000000000000000 P 941b88a30258457b9ab7c2d3101ca378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 941b88a30258457b9ab7c2d3101ca378, State: Initialized, Role: FOLLOWER
I20260811 22:26:40.710876 15384 consensus_queue.cc:260] T 00000000000000000000000000000000 P 941b88a30258457b9ab7c2d3101ca378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "941b88a30258457b9ab7c2d3101ca378" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 41115 } }
I20260811 22:26:40.711091 15384 raft_consensus.cc:388] T 00000000000000000000000000000000 P 941b88a30258457b9ab7c2d3101ca378 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:26:40.711223 15384 raft_consensus.cc:482] T 00000000000000000000000000000000 P 941b88a30258457b9ab7c2d3101ca378 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:26:40.711396 15384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 941b88a30258457b9ab7c2d3101ca378 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:26:40.713382 15384 raft_consensus.cc:504] T 00000000000000000000000000000000 P 941b88a30258457b9ab7c2d3101ca378 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "941b88a30258457b9ab7c2d3101ca378" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 41115 } }
I20260811 22:26:40.713829 15384 leader_election.cc:302] T 00000000000000000000000000000000 P 941b88a30258457b9ab7c2d3101ca378 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 941b88a30258457b9ab7c2d3101ca378; no voters:
I20260811 22:26:40.714352 15384 leader_election.cc:288] T 00000000000000000000000000000000 P 941b88a30258457b9ab7c2d3101ca378 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:26:40.714465 15394 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 941b88a30258457b9ab7c2d3101ca378 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:26:40.715106 15394 raft_consensus.cc:686] T 00000000000000000000000000000000 P 941b88a30258457b9ab7c2d3101ca378 [term 1 LEADER]: Becoming Leader. State: Replica: 941b88a30258457b9ab7c2d3101ca378, State: Running, Role: LEADER
I20260811 22:26:40.715634 15394 consensus_queue.cc:237] T 00000000000000000000000000000000 P 941b88a30258457b9ab7c2d3101ca378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "941b88a30258457b9ab7c2d3101ca378" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 41115 } }
I20260811 22:26:40.716123 15384 sys_catalog.cc:564] T 00000000000000000000000000000000 P 941b88a30258457b9ab7c2d3101ca378 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:26:40.717944 15397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 941b88a30258457b9ab7c2d3101ca378 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 941b88a30258457b9ab7c2d3101ca378. Latest consensus state: current_term: 1 leader_uuid: "941b88a30258457b9ab7c2d3101ca378" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "941b88a30258457b9ab7c2d3101ca378" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 41115 } } }
I20260811 22:26:40.718246 15397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 941b88a30258457b9ab7c2d3101ca378 [sys.catalog]: This master's current role is: LEADER
I20260811 22:26:40.719686 15401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:26:40.721170 15395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 941b88a30258457b9ab7c2d3101ca378 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "941b88a30258457b9ab7c2d3101ca378" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "941b88a30258457b9ab7c2d3101ca378" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 41115 } } }
I20260811 22:26:40.721438 15395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 941b88a30258457b9ab7c2d3101ca378 [sys.catalog]: This master's current role is: LEADER
I20260811 22:26:40.724208 15401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:26:40.736357 15401 catalog_manager.cc:1347] Generated new cluster ID: 12437767cd104357b8d1c9cda8fbc148
I20260811 22:26:40.736475 15401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:26:40.750797 15401 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:26:40.751979 15401 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:26:40.762108 15401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 941b88a30258457b9ab7c2d3101ca378: Generated new TSK 0
I20260811 22:26:40.762753 15401 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:26:40.771760 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.1:0
--local_ip_for_outbound_sockets=127.29.106.1
--webserver_interface=127.29.106.1
--webserver_port=0
--tserver_master_addrs=127.29.106.62:41115
--builtin_ntp_servers=127.29.106.20:41783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260811 22:26:41.099205 15420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:26:41.099506 15420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:26:41.099627 15420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:26:41.106518 15420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:26:41.106602 15420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:26:41.106737 15420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.1
I20260811 22:26:41.115389 15420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:41783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.106.1
--webserver_port=0
--tserver_master_addrs=127.29.106.62:41115
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.1
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:26:41.116648 15420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:26:41.118445 15420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:26:41.130793 15430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:41.130861 15433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:41.130829 15428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:26:41.131853 15420 server_base.cc:1034] running on GCE node
I20260811 22:26:41.132402 15420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:26:41.133271 15420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:26:41.134465 15420 hybrid_clock.cc:648] HybridClock initialized: now 1786487201134408 us; error 49 us; skew 500 ppm
I20260811 22:26:41.134760 15420 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 22:26:41.136873 15420 webserver.cc:460] Webserver started at http://127.29.106.1:42065/ using document root <none> and password file <none>
I20260811 22:26:41.137507 15420 fs_manager.cc:359] Metadata directory not provided
I20260811 22:26:41.137616 15420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:26:41.137928 15420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:26:41.140689 15420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/data/instance:
uuid: "cb919e32cc9c40139b5eae676d12b71a"
format_stamp: "Formatted at 2026-08-11 22:26:41 on dist-test-slave-gjrz"
server_key: "abf60583c98018c71bb172a04d7e7ba8"
server_key_iv: "882fcfe36a43b339d4c191c3fddfc0a4"
server_key_version: "encryptionkey@0"
I20260811 22:26:41.141669 15420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance:
uuid: "cb919e32cc9c40139b5eae676d12b71a"
format_stamp: "Formatted at 2026-08-11 22:26:41 on dist-test-slave-gjrz"
server_key: "abf60583c98018c71bb172a04d7e7ba8"
server_key_iv: "882fcfe36a43b339d4c191c3fddfc0a4"
server_key_version: "encryptionkey@0"
I20260811 22:26:41.146721 15420 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260811 22:26:41.150058 15443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:41.151239 15420 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 22:26:41.151444 15420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/wal
uuid: "cb919e32cc9c40139b5eae676d12b71a"
format_stamp: "Formatted at 2026-08-11 22:26:41 on dist-test-slave-gjrz"
server_key: "abf60583c98018c71bb172a04d7e7ba8"
server_key_iv: "882fcfe36a43b339d4c191c3fddfc0a4"
server_key_version: "encryptionkey@0"
I20260811 22:26:41.151648 15420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:26:41.167174 15420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:26:41.167918 15420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:26:41.168177 15420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:26:41.169008 15420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:26:41.170331 15420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:26:41.170425 15420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:41.170527 15420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:26:41.170604 15420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:41.198764 15420 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.1:39085
I20260811 22:26:41.198813 15647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.1:39085 every 8 connection(s)
I20260811 22:26:41.200029 15420 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
I20260811 22:26:41.207190 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 15420
I20260811 22:26:41.207399 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance
I20260811 22:26:41.207994 30120 external_mini_cluster.cc:1428] Setting key 81dc2fa9e3aa32ed319b588a67545182
I20260811 22:26:41.210817 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.2:0
--local_ip_for_outbound_sockets=127.29.106.2
--webserver_interface=127.29.106.2
--webserver_port=0
--tserver_master_addrs=127.29.106.62:41115
--builtin_ntp_servers=127.29.106.20:41783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260811 22:26:41.212262 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:41.202500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:45795 (local address 127.29.106.62:41115)
0811 22:26:41.202923 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:41.202928 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:41.204020 (+ 1092us) server_negotiation.cc:407] Connection header received
0811 22:26:41.205716 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:41.205721 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:41.205797 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:41.205890 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:41.206916 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:41.207733 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:41.209548 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:41.210105 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:41.210232 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:41.211456 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:41.211462 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:41.211467 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:26:41.211489 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:41.211556 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:26:41.211574 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:41.211668 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:41.211828 (+ 160us) server_negotiation.cc:299] Negotiation successful
0811 22:26:41.211925 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":227,"threads_started":1}
I20260811 22:26:41.213091 15652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:41.202988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:41115 (local address 127.29.106.1:45795)
0811 22:26:41.203861 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:41.203906 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:41.205454 (+ 1548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:41.206126 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:41.206134 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:41.206213 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:41.206752 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:41.206770 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:41.207891 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:41.207895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:41.209422 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:41.209436 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:41.210310 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:41.210315 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:41.210507 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:41.211142 (+ 635us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 22:26:41.211145 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:41.211180 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 22:26:41.211249 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:26:41.211663 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:26:41.211669 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:26:41.211675 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 22:26:41.211804 (+ 129us) client_negotiation.cc:240] Negotiation successful
0811 22:26:41.212063 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":280,"threads_started":1}
I20260811 22:26:41.214417 15649 heartbeater.cc:347] Connected to a master server at 127.29.106.62:41115
I20260811 22:26:41.214715 15649 heartbeater.cc:464] Registering TS with master...
I20260811 22:26:41.215598 15649 heartbeater.cc:511] Master 127.29.106.62:41115 requested a full tablet report, sending...
I20260811 22:26:41.217666 15328 ts_manager.cc:194] Registered new tserver with Master: cb919e32cc9c40139b5eae676d12b71a (127.29.106.1:39085)
I20260811 22:26:41.219141 15328 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.106.1:45795
W20260811 22:26:41.537631 15653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:26:41.537868 15653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:26:41.537989 15653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:26:41.544942 15653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:26:41.545032 15653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:26:41.545181 15653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.2
I20260811 22:26:41.553869 15653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:41783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.106.2
--webserver_port=0
--tserver_master_addrs=127.29.106.62:41115
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.2
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:26:41.555187 15653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:26:41.557322 15653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:26:41.569516 15660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:41.569482 15662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:41.569551 15659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:26:41.570259 15653 server_base.cc:1034] running on GCE node
I20260811 22:26:41.570775 15653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:26:41.571645 15653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:26:41.572836 15653 hybrid_clock.cc:648] HybridClock initialized: now 1786487201572787 us; error 37 us; skew 500 ppm
I20260811 22:26:41.573119 15653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 22:26:41.575110 15653 webserver.cc:460] Webserver started at http://127.29.106.2:34451/ using document root <none> and password file <none>
I20260811 22:26:41.575780 15653 fs_manager.cc:359] Metadata directory not provided
I20260811 22:26:41.575899 15653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:26:41.576218 15653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:26:41.578827 15653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/data/instance:
uuid: "43dece54dd38437a87e5474bd4656e9b"
format_stamp: "Formatted at 2026-08-11 22:26:41 on dist-test-slave-gjrz"
server_key: "6c8209c518adc35311ee9070566e5cf9"
server_key_iv: "e1756033276895d9b7d2510c819aefcd"
server_key_version: "encryptionkey@0"
I20260811 22:26:41.579622 15653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance:
uuid: "43dece54dd38437a87e5474bd4656e9b"
format_stamp: "Formatted at 2026-08-11 22:26:41 on dist-test-slave-gjrz"
server_key: "6c8209c518adc35311ee9070566e5cf9"
server_key_iv: "e1756033276895d9b7d2510c819aefcd"
server_key_version: "encryptionkey@0"
I20260811 22:26:41.584343 15653 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260811 22:26:41.587519 15678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:41.588653 15653 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:26:41.588790 15653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/wal
uuid: "43dece54dd38437a87e5474bd4656e9b"
format_stamp: "Formatted at 2026-08-11 22:26:41 on dist-test-slave-gjrz"
server_key: "6c8209c518adc35311ee9070566e5cf9"
server_key_iv: "e1756033276895d9b7d2510c819aefcd"
server_key_version: "encryptionkey@0"
I20260811 22:26:41.588974 15653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:26:41.608719 15653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:26:41.609421 15653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:26:41.609691 15653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:26:41.610538 15653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:26:41.611850 15653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:26:41.611953 15653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:41.612051 15653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:26:41.612125 15653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:41.641752 15653 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.2:32899
I20260811 22:26:41.641811 15860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.2:32899 every 8 connection(s)
I20260811 22:26:41.642973 15653 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
I20260811 22:26:41.647086 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 15653
I20260811 22:26:41.647290 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance
I20260811 22:26:41.647730 30120 external_mini_cluster.cc:1428] Setting key 46a823ef3287e9793bc4ba5a7c4476d3
I20260811 22:26:41.650257 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.3:0
--local_ip_for_outbound_sockets=127.29.106.3
--webserver_interface=127.29.106.3
--webserver_port=0
--tserver_master_addrs=127.29.106.62:41115
--builtin_ntp_servers=127.29.106.20:41783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260811 22:26:41.654951 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:41.645386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:41771 (local address 127.29.106.62:41115)
0811 22:26:41.645566 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:41.645571 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:41.646900 (+ 1329us) server_negotiation.cc:407] Connection header received
0811 22:26:41.648938 (+ 2038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:41.648944 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:41.649026 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:41.649111 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:41.650224 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:41.650976 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:41.652513 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:41.652913 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:41.653022 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:41.654225 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:41.654231 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:41.654236 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:26:41.654252 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:41.654282 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:26:41.654294 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:41.654374 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:41.654621 (+ 247us) server_negotiation.cc:299] Negotiation successful
0811 22:26:41.654706 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260811 22:26:41.656152 15864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:41.645842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:41115 (local address 127.29.106.2:41771)
0811 22:26:41.646765 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:41.646800 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:41.648692 (+ 1892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:41.649359 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:41.649371 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:41.649480 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:41.650069 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:41.650086 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:41.651122 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:41.651125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:41.652376 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:41.652388 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:41.653116 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:41.653124 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:41.653325 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:41.653920 (+ 595us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 22:26:41.653923 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:41.653985 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 22:26:41.654060 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:26:41.654419 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:26:41.654429 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:26:41.654437 (+ 8us) client_negotiation.cc:769] Sending connection context
0811 22:26:41.654617 (+ 180us) client_negotiation.cc:240] Negotiation successful
0811 22:26:41.654954 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":254,"threads_started":1}
I20260811 22:26:41.657361 15861 heartbeater.cc:347] Connected to a master server at 127.29.106.62:41115
I20260811 22:26:41.657634 15861 heartbeater.cc:464] Registering TS with master...
I20260811 22:26:41.658329 15861 heartbeater.cc:511] Master 127.29.106.62:41115 requested a full tablet report, sending...
I20260811 22:26:41.659672 15328 ts_manager.cc:194] Registered new tserver with Master: 43dece54dd38437a87e5474bd4656e9b (127.29.106.2:32899)
I20260811 22:26:41.660439 15328 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.106.2:41771
W20260811 22:26:41.972818 15865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:26:41.973071 15865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:26:41.973177 15865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:26:41.980185 15865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:26:41.980271 15865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:26:41.980499 15865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.3
I20260811 22:26:41.989362 15865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:41783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.106.3
--webserver_port=0
--tserver_master_addrs=127.29.106.62:41115
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.3
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:26:41.990476 15865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:26:41.992275 15865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:26:42.004683 15870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:42.004771 15877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:42.004794 15873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:26:42.005488 15865 server_base.cc:1034] running on GCE node
I20260811 22:26:42.006037 15865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:26:42.006943 15865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:26:42.008174 15865 hybrid_clock.cc:648] HybridClock initialized: now 1786487202008109 us; error 48 us; skew 500 ppm
I20260811 22:26:42.008494 15865 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 22:26:42.010610 15865 webserver.cc:460] Webserver started at http://127.29.106.3:43731/ using document root <none> and password file <none>
I20260811 22:26:42.011255 15865 fs_manager.cc:359] Metadata directory not provided
I20260811 22:26:42.011399 15865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:26:42.011721 15865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:26:42.014416 15865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/data/instance:
uuid: "a063d7681fde484aa646c2be3cd63d76"
format_stamp: "Formatted at 2026-08-11 22:26:42 on dist-test-slave-gjrz"
server_key: "1d2a1f83ffd065813e917e99cddaf8eb"
server_key_iv: "ab7a48447b9dba57be22e75e65c4515b"
server_key_version: "encryptionkey@0"
I20260811 22:26:42.015187 15865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance:
uuid: "a063d7681fde484aa646c2be3cd63d76"
format_stamp: "Formatted at 2026-08-11 22:26:42 on dist-test-slave-gjrz"
server_key: "1d2a1f83ffd065813e917e99cddaf8eb"
server_key_iv: "ab7a48447b9dba57be22e75e65c4515b"
server_key_version: "encryptionkey@0"
I20260811 22:26:42.020287 15865 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260811 22:26:42.023605 15892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:42.024847 15865 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260811 22:26:42.025007 15865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/wal
uuid: "a063d7681fde484aa646c2be3cd63d76"
format_stamp: "Formatted at 2026-08-11 22:26:42 on dist-test-slave-gjrz"
server_key: "1d2a1f83ffd065813e917e99cddaf8eb"
server_key_iv: "ab7a48447b9dba57be22e75e65c4515b"
server_key_version: "encryptionkey@0"
I20260811 22:26:42.025198 15865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:26:42.043565 15865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:26:42.044335 15865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:26:42.044612 15865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:26:42.045495 15865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:26:42.046891 15865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:26:42.046998 15865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:42.047104 15865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:26:42.047183 15865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:42.077659 15865 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.3:37035
I20260811 22:26:42.077704 16079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.3:37035 every 8 connection(s)
I20260811 22:26:42.078893 15865 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
I20260811 22:26:42.085839 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 15865
I20260811 22:26:42.086025 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance
I20260811 22:26:42.086434 30120 external_mini_cluster.cc:1428] Setting key 370035a9d5fa4fab14bb54b3e7f0d2c1
I20260811 22:26:42.091120 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:42.081266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:59049 (local address 127.29.106.62:41115)
0811 22:26:42.081450 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:42.081455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:42.082780 (+ 1325us) server_negotiation.cc:407] Connection header received
0811 22:26:42.084554 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:42.084561 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:42.084636 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:42.084722 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:42.085920 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:42.086740 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:42.088531 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:42.088962 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:42.089075 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:42.090319 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:42.090324 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:42.090327 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:26:42.090344 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:42.090374 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:26:42.090385 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:42.090465 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:42.090667 (+ 202us) server_negotiation.cc:299] Negotiation successful
0811 22:26:42.090765 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260811 22:26:42.092229 16085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:42.081720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:41115 (local address 127.29.106.3:59049)
0811 22:26:42.082618 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:42.082662 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:42.084272 (+ 1610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:42.084984 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:42.084997 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:42.085079 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:42.085730 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:42.085757 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:42.086932 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:42.086937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:42.088378 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:42.088395 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:42.089172 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:42.089179 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:42.089473 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:42.090042 (+ 569us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 22:26:42.090044 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:42.090082 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 22:26:42.090151 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:26:42.090478 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:26:42.090489 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:26:42.090496 (+ 7us) client_negotiation.cc:769] Sending connection context
0811 22:26:42.090641 (+ 145us) client_negotiation.cc:240] Negotiation successful
0811 22:26:42.091019 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":283,"threads_started":1}
I20260811 22:26:42.093418 16080 heartbeater.cc:347] Connected to a master server at 127.29.106.62:41115
I20260811 22:26:42.093677 16080 heartbeater.cc:464] Registering TS with master...
I20260811 22:26:42.094380 16080 heartbeater.cc:511] Master 127.29.106.62:41115 requested a full tablet report, sending...
I20260811 22:26:42.095633 15328 ts_manager.cc:194] Registered new tserver with Master: a063d7681fde484aa646c2be3cd63d76 (127.29.106.3:37035)
I20260811 22:26:42.096391 15328 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.106.3:59049
I20260811 22:26:42.101208 30120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260811 22:26:42.106936 15651 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:42.105355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56912 (local address 127.29.106.62:41115)
0811 22:26:42.105488 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:42.105492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:42.105980 (+ 488us) server_negotiation.cc:407] Connection header received
0811 22:26:42.106139 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:42.106143 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:42.106211 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:42.106290 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:42.106711 (+ 421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56912: BlockingRecv error: recv got EOF from 127.0.0.1:56912 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260811 22:26:42.107542 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 15420
I20260811 22:26:42.114372 16096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:42.108418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48212 (local address 127.29.106.1:39085)
0811 22:26:42.108906 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:42.108913 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:42.108938 (+ 25us) server_negotiation.cc:407] Connection header received
0811 22:26:42.108995 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:42.109001 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:42.109168 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:42.109289 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:42.109570 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:42.110539 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:42.111733 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:42.112250 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:42.112347 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:42.113175 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:42.113183 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:42.113195 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:26:42.113729 (+ 534us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:42.113778 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:26:42.113797 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:42.113880 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:42.113978 (+ 98us) server_negotiation.cc:299] Negotiation successful
0811 22:26:42.114098 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":242,"threads_started":1}
I20260811 22:26:42.115500 15642 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:26:42.196535 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 15420
I20260811 22:26:42.215682 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 15653
I20260811 22:26:42.223184 16098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:42.216581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44174 (local address 127.29.106.2:32899)
0811 22:26:42.217143 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:42.217150 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:42.217179 (+ 29us) server_negotiation.cc:407] Connection header received
0811 22:26:42.217249 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:42.217255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:42.217469 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:42.217598 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:42.217992 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:42.219003 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:42.220275 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:42.220824 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:42.220918 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:42.221837 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:42.221845 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:42.221859 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:26:42.222449 (+ 590us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:42.222502 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:26:42.222522 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:42.222610 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:42.222757 (+ 147us) server_negotiation.cc:299] Negotiation successful
0811 22:26:42.222886 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":252,"threads_started":1}
I20260811 22:26:42.224436 15852 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:26:42.305055 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 15653
I20260811 22:26:42.330116 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 15865
I20260811 22:26:42.338279 16102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:42.331057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57570 (local address 127.29.106.3:37035)
0811 22:26:42.331711 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:42.331718 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:42.331748 (+ 30us) server_negotiation.cc:407] Connection header received
0811 22:26:42.331809 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:42.331815 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:42.332062 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:42.332204 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:42.332685 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:42.333754 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:42.334986 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:42.335715 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:42.335833 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:42.336866 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:42.336875 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:42.336888 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:26:42.337469 (+ 581us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:42.337518 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:26:42.337538 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:42.337640 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:42.337793 (+ 153us) server_negotiation.cc:299] Negotiation successful
0811 22:26:42.337941 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":345,"threads_started":1}
I20260811 22:26:42.339531 16072 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:26:42.419488 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 15865
I20260811 22:26:42.443483 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 15269
I20260811 22:26:42.444679 15374 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:26:42.546661 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 15269
2026-08-11T22:26:42Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2360 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-11T22:26:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:26:42Z Disabled control of system clock
I20260811 22:26:42.629016 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:40113
--webserver_interface=127.29.106.62
--webserver_port=0
--builtin_ntp_servers=127.29.106.20:42875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:40113
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260811 22:26:42.949440 16116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:26:42.949724 16116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:26:42.949852 16116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:26:42.956831 16116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:26:42.956918 16116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:26:42.956991 16116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:26:42.957067 16116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:26:42.957129 16116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:26:42.965816 16116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:42875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:40113
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:40113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.106.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:26:42.966977 16116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:26:42.968696 16116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:26:42.979770 16125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:42.979682 16126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:42.979812 16130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:26:42.980372 16116 server_base.cc:1034] running on GCE node
I20260811 22:26:42.981093 16116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:26:42.982342 16116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:26:42.983534 16116 hybrid_clock.cc:648] HybridClock initialized: now 1786487202983494 us; error 32 us; skew 500 ppm
I20260811 22:26:42.983846 16116 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 22:26:42.985870 16116 webserver.cc:460] Webserver started at http://127.29.106.62:46415/ using document root <none> and password file <none>
I20260811 22:26:42.986507 16116 fs_manager.cc:359] Metadata directory not provided
I20260811 22:26:42.986603 16116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:26:42.986898 16116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:26:42.989629 16116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "58d5e5bfd9854661b18a55622b093b23"
format_stamp: "Formatted at 2026-08-11 22:26:42 on dist-test-slave-gjrz"
server_key: "d5ce13fd516ee3cddc28e60d30e10e63"
server_key_iv: "e6507c8676f455bbfcb6f4de5922b82e"
server_key_version: "encryptionkey@0"
I20260811 22:26:42.990374 16116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "58d5e5bfd9854661b18a55622b093b23"
format_stamp: "Formatted at 2026-08-11 22:26:42 on dist-test-slave-gjrz"
server_key: "d5ce13fd516ee3cddc28e60d30e10e63"
server_key_iv: "e6507c8676f455bbfcb6f4de5922b82e"
server_key_version: "encryptionkey@0"
I20260811 22:26:42.995092 16116 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260811 22:26:42.998231 16142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:42.999490 16116 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:26:42.999632 16116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "58d5e5bfd9854661b18a55622b093b23"
format_stamp: "Formatted at 2026-08-11 22:26:42 on dist-test-slave-gjrz"
server_key: "d5ce13fd516ee3cddc28e60d30e10e63"
server_key_iv: "e6507c8676f455bbfcb6f4de5922b82e"
server_key_version: "encryptionkey@0"
I20260811 22:26:42.999814 16116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:26:43.005184 16116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:26:43.005877 16116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:26:43.006142 16116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:26:43.023007 16116 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.62:40113
I20260811 22:26:43.023049 16233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.62:40113 every 8 connection(s)
I20260811 22:26:43.024406 16116 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:26:43.028163 16234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:26:43.028359 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 16116
I20260811 22:26:43.028532 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:26:43.029171 30120 external_mini_cluster.cc:1428] Setting key ffe439d77b44c9e7f602cc271acb2449
I20260811 22:26:43.037387 16237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:43.031079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44992 (local address 127.29.106.62:40113)
0811 22:26:43.032064 (+ 985us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:43.032074 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:43.032118 (+ 44us) server_negotiation.cc:407] Connection header received
0811 22:26:43.034030 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:43.034052 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:43.034453 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:43.034819 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:43.035844 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:43.035852 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:43.035864 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:26:43.035879 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:43.035939 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:26:43.035966 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:43.036170 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:43.036304 (+ 134us) server_negotiation.cc:299] Negotiation successful
0811 22:26:43.036532 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":373,"threads_started":1}
I20260811 22:26:43.040748 16234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58d5e5bfd9854661b18a55622b093b23: Bootstrap starting.
I20260811 22:26:43.043597 16234 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58d5e5bfd9854661b18a55622b093b23: Neither blocks nor log segments found. Creating new log.
I20260811 22:26:43.044718 16234 log.cc:824] T 00000000000000000000000000000000 P 58d5e5bfd9854661b18a55622b093b23: Log is configured to *not* fsync() on all Append() calls
I20260811 22:26:43.047236 16234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58d5e5bfd9854661b18a55622b093b23: No bootstrap required, opened a new log
I20260811 22:26:43.054490 16234 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58d5e5bfd9854661b18a55622b093b23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58d5e5bfd9854661b18a55622b093b23" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 40113 } }
I20260811 22:26:43.054824 16234 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58d5e5bfd9854661b18a55622b093b23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:26:43.054912 16234 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58d5e5bfd9854661b18a55622b093b23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d5e5bfd9854661b18a55622b093b23, State: Initialized, Role: FOLLOWER
I20260811 22:26:43.055490 16234 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58d5e5bfd9854661b18a55622b093b23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58d5e5bfd9854661b18a55622b093b23" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 40113 } }
I20260811 22:26:43.055711 16234 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58d5e5bfd9854661b18a55622b093b23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:26:43.055832 16234 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58d5e5bfd9854661b18a55622b093b23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:26:43.055979 16234 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58d5e5bfd9854661b18a55622b093b23 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:26:43.058185 16234 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58d5e5bfd9854661b18a55622b093b23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58d5e5bfd9854661b18a55622b093b23" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 40113 } }
I20260811 22:26:43.058662 16234 leader_election.cc:302] T 00000000000000000000000000000000 P 58d5e5bfd9854661b18a55622b093b23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58d5e5bfd9854661b18a55622b093b23; no voters:
I20260811 22:26:43.059242 16234 leader_election.cc:288] T 00000000000000000000000000000000 P 58d5e5bfd9854661b18a55622b093b23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:26:43.059414 16240 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58d5e5bfd9854661b18a55622b093b23 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:26:43.060086 16240 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58d5e5bfd9854661b18a55622b093b23 [term 1 LEADER]: Becoming Leader. State: Replica: 58d5e5bfd9854661b18a55622b093b23, State: Running, Role: LEADER
I20260811 22:26:43.060595 16240 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58d5e5bfd9854661b18a55622b093b23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58d5e5bfd9854661b18a55622b093b23" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 40113 } }
I20260811 22:26:43.061189 16234 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58d5e5bfd9854661b18a55622b093b23 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:26:43.063091 16246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58d5e5bfd9854661b18a55622b093b23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58d5e5bfd9854661b18a55622b093b23. Latest consensus state: current_term: 1 leader_uuid: "58d5e5bfd9854661b18a55622b093b23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58d5e5bfd9854661b18a55622b093b23" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 40113 } } }
I20260811 22:26:43.063521 16246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58d5e5bfd9854661b18a55622b093b23 [sys.catalog]: This master's current role is: LEADER
I20260811 22:26:43.064543 16242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58d5e5bfd9854661b18a55622b093b23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58d5e5bfd9854661b18a55622b093b23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58d5e5bfd9854661b18a55622b093b23" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 40113 } } }
I20260811 22:26:43.064929 16242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58d5e5bfd9854661b18a55622b093b23 [sys.catalog]: This master's current role is: LEADER
I20260811 22:26:43.065701 16258 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:26:43.071216 16258 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:26:43.079676 16258 catalog_manager.cc:1347] Generated new cluster ID: ed8d1a6c6237452c85e9904d1b8e53ae
I20260811 22:26:43.079773 16258 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:26:43.126956 16258 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:26:43.127409 16258 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:26:43.132808 16258 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58d5e5bfd9854661b18a55622b093b23: Generated new TSK 0
I20260811 22:26:43.133494 16258 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:26:43.144996 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.1:0
--local_ip_for_outbound_sockets=127.29.106.1
--webserver_interface=127.29.106.1
--webserver_port=0
--tserver_master_addrs=127.29.106.62:40113
--builtin_ntp_servers=127.29.106.20:42875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260811 22:26:43.465670 16273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:26:43.465916 16273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:26:43.466050 16273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:26:43.472990 16273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:26:43.473080 16273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:26:43.473220 16273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.1
I20260811 22:26:43.481781 16273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:42875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.106.1
--webserver_port=0
--tserver_master_addrs=127.29.106.62:40113
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.1
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:26:43.482914 16273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:26:43.484572 16273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:26:43.498030 16282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:43.498425 16284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:43.498613 16287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:26:43.499075 16273 server_base.cc:1034] running on GCE node
I20260811 22:26:43.499675 16273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:26:43.500541 16273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:26:43.501753 16273 hybrid_clock.cc:648] HybridClock initialized: now 1786487203501704 us; error 38 us; skew 500 ppm
I20260811 22:26:43.502069 16273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 22:26:43.504212 16273 webserver.cc:460] Webserver started at http://127.29.106.1:38303/ using document root <none> and password file <none>
I20260811 22:26:43.504943 16273 fs_manager.cc:359] Metadata directory not provided
I20260811 22:26:43.505052 16273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:26:43.505399 16273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:26:43.510758 16273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/data/instance:
uuid: "613d8b2c62354e34b8fa5223e66d5ab0"
format_stamp: "Formatted at 2026-08-11 22:26:43 on dist-test-slave-gjrz"
server_key: "c72a552c684d3bebd6b893e2abdda476"
server_key_iv: "cbe78b2ec52906c09e82e4275f3cab6c"
server_key_version: "encryptionkey@0"
I20260811 22:26:43.511559 16273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance:
uuid: "613d8b2c62354e34b8fa5223e66d5ab0"
format_stamp: "Formatted at 2026-08-11 22:26:43 on dist-test-slave-gjrz"
server_key: "c72a552c684d3bebd6b893e2abdda476"
server_key_iv: "cbe78b2ec52906c09e82e4275f3cab6c"
server_key_version: "encryptionkey@0"
I20260811 22:26:43.516242 16273 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260811 22:26:43.519381 16295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:43.520659 16273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260811 22:26:43.520807 16273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/wal
uuid: "613d8b2c62354e34b8fa5223e66d5ab0"
format_stamp: "Formatted at 2026-08-11 22:26:43 on dist-test-slave-gjrz"
server_key: "c72a552c684d3bebd6b893e2abdda476"
server_key_iv: "cbe78b2ec52906c09e82e4275f3cab6c"
server_key_version: "encryptionkey@0"
I20260811 22:26:43.520996 16273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:26:43.526242 16273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:26:43.526952 16273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:26:43.527189 16273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:26:43.528066 16273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:26:43.529338 16273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:26:43.529430 16273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:43.529538 16273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:26:43.529606 16273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:43.558779 16273 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.1:37077
I20260811 22:26:43.558838 16497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.1:37077 every 8 connection(s)
I20260811 22:26:43.560216 16273 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
I20260811 22:26:43.560806 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 16273
I20260811 22:26:43.560992 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance
I20260811 22:26:43.561641 30120 external_mini_cluster.cc:1428] Setting key ed007f06426711c1fc92b9c881f78e5c
I20260811 22:26:43.564908 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.2:0
--local_ip_for_outbound_sockets=127.29.106.2
--webserver_interface=127.29.106.2
--webserver_port=0
--tserver_master_addrs=127.29.106.62:40113
--builtin_ntp_servers=127.29.106.20:42875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260811 22:26:43.570568 16503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:43.564474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:47745 (local address 127.29.106.62:40113)
0811 22:26:43.565132 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:43.565138 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:43.566388 (+ 1250us) server_negotiation.cc:407] Connection header received
0811 22:26:43.568524 (+ 2136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:43.568529 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:43.568580 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:43.568667 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:43.569978 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:43.569982 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:43.569985 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:26:43.570000 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:43.570047 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:26:43.570059 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:43.570142 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:43.570320 (+ 178us) server_negotiation.cc:299] Negotiation successful
0811 22:26:43.570366 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":424,"threads_started":1}
I20260811 22:26:43.571717 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:43.565153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:40113 (local address 127.29.106.1:47745)
0811 22:26:43.566234 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:43.566279 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:43.568257 (+ 1978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:43.568911 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:43.568923 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:43.569031 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:43.569660 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 22:26:43.569663 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:43.569707 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 22:26:43.569809 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:26:43.570139 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:26:43.570150 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:26:43.570156 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 22:26:43.570405 (+ 249us) client_negotiation.cc:240] Negotiation successful
0811 22:26:43.570644 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":297,"threads_started":1}
I20260811 22:26:43.573014 16498 heartbeater.cc:347] Connected to a master server at 127.29.106.62:40113
I20260811 22:26:43.573283 16498 heartbeater.cc:464] Registering TS with master...
I20260811 22:26:43.573805 16498 heartbeater.cc:511] Master 127.29.106.62:40113 requested a full tablet report, sending...
I20260811 22:26:43.575714 16164 ts_manager.cc:194] Registered new tserver with Master: 613d8b2c62354e34b8fa5223e66d5ab0 (127.29.106.1:37077)
W20260811 22:26:43.885643 16506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:26:43.885885 16506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:26:43.885999 16506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:26:43.893016 16506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:26:43.893103 16506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:26:43.893247 16506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.2
I20260811 22:26:43.901913 16506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:42875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.106.2
--webserver_port=0
--tserver_master_addrs=127.29.106.62:40113
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.2
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:26:43.903134 16506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:26:43.905023 16506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:26:43.917078 16516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:43.917122 16523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:43.917166 16518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:26:43.917987 16506 server_base.cc:1034] running on GCE node
I20260811 22:26:43.918519 16506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:26:43.919410 16506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:26:43.920630 16506 hybrid_clock.cc:648] HybridClock initialized: now 1786487203920566 us; error 47 us; skew 500 ppm
I20260811 22:26:43.920941 16506 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 22:26:43.923075 16506 webserver.cc:460] Webserver started at http://127.29.106.2:36885/ using document root <none> and password file <none>
I20260811 22:26:43.923736 16506 fs_manager.cc:359] Metadata directory not provided
I20260811 22:26:43.923837 16506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:26:43.924132 16506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:26:43.926810 16506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/data/instance:
uuid: "c1f511aee3234f4e9fc708ad66adbb66"
format_stamp: "Formatted at 2026-08-11 22:26:43 on dist-test-slave-gjrz"
server_key: "7ef4cfa79062f6ce639a6d944bab3899"
server_key_iv: "0c60c633ad125fc3a6a2bfa4d04ed0df"
server_key_version: "encryptionkey@0"
I20260811 22:26:43.927600 16506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance:
uuid: "c1f511aee3234f4e9fc708ad66adbb66"
format_stamp: "Formatted at 2026-08-11 22:26:43 on dist-test-slave-gjrz"
server_key: "7ef4cfa79062f6ce639a6d944bab3899"
server_key_iv: "0c60c633ad125fc3a6a2bfa4d04ed0df"
server_key_version: "encryptionkey@0"
I20260811 22:26:43.932260 16506 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 22:26:43.935321 16530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:43.936553 16506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260811 22:26:43.936715 16506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/wal
uuid: "c1f511aee3234f4e9fc708ad66adbb66"
format_stamp: "Formatted at 2026-08-11 22:26:43 on dist-test-slave-gjrz"
server_key: "7ef4cfa79062f6ce639a6d944bab3899"
server_key_iv: "0c60c633ad125fc3a6a2bfa4d04ed0df"
server_key_version: "encryptionkey@0"
I20260811 22:26:43.936918 16506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:26:43.942267 16506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:26:43.942977 16506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:26:43.943219 16506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:26:43.944160 16506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:26:43.945590 16506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:26:43.945689 16506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:43.945793 16506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:26:43.945863 16506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:43.973635 16506 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.2:42651
I20260811 22:26:43.973696 16731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.2:42651 every 8 connection(s)
I20260811 22:26:43.974812 16506 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
I20260811 22:26:43.982229 16503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:43.977191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:55073 (local address 127.29.106.62:40113)
0811 22:26:43.977417 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:43.977424 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:43.978734 (+ 1310us) server_negotiation.cc:407] Connection header received
0811 22:26:43.980328 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:43.980332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:43.980385 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:43.980468 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:43.981568 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:43.981572 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:43.981575 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:26:43.981591 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:43.981623 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:26:43.981634 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:43.981724 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:43.981852 (+ 128us) server_negotiation.cc:299] Negotiation successful
0811 22:26:43.981931 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260811 22:26:43.982275 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 16506
I20260811 22:26:43.982427 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance
I20260811 22:26:43.982851 30120 external_mini_cluster.cc:1428] Setting key 54dee58dba48dce449b047be618112b3
I20260811 22:26:43.983079 16736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:43.977720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:40113 (local address 127.29.106.2:55073)
0811 22:26:43.978590 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:43.978626 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:43.980105 (+ 1479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:43.980641 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:43.980648 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:43.980731 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:43.981297 (+ 566us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 22:26:43.981299 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:43.981334 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 22:26:43.981413 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:26:43.981704 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:26:43.981709 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:26:43.981715 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 22:26:43.981919 (+ 204us) client_negotiation.cc:240] Negotiation successful
0811 22:26:43.982103 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":255,"threads_started":1}
I20260811 22:26:43.984423 16733 heartbeater.cc:347] Connected to a master server at 127.29.106.62:40113
I20260811 22:26:43.984678 16733 heartbeater.cc:464] Registering TS with master...
I20260811 22:26:43.985227 16733 heartbeater.cc:511] Master 127.29.106.62:40113 requested a full tablet report, sending...
I20260811 22:26:43.985795 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.3:0
--local_ip_for_outbound_sockets=127.29.106.3
--webserver_interface=127.29.106.3
--webserver_port=0
--tserver_master_addrs=127.29.106.62:40113
--builtin_ntp_servers=127.29.106.20:42875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260811 22:26:43.986552 16163 ts_manager.cc:194] Registered new tserver with Master: c1f511aee3234f4e9fc708ad66adbb66 (127.29.106.2:42651)
W20260811 22:26:44.309131 16739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:26:44.309377 16739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:26:44.309463 16739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:26:44.316383 16739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:26:44.316469 16739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:26:44.316579 16739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.3
I20260811 22:26:44.325335 16739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:42875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.106.3
--webserver_port=0
--tserver_master_addrs=127.29.106.62:40113
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.3
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:26:44.326468 16739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:26:44.328194 16739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:26:44.341043 16748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:44.340976 16749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:44.341094 16752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:26:44.342144 16739 server_base.cc:1034] running on GCE node
I20260811 22:26:44.342711 16739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:26:44.343638 16739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:26:44.344833 16739 hybrid_clock.cc:648] HybridClock initialized: now 1786487204344780 us; error 43 us; skew 500 ppm
I20260811 22:26:44.345126 16739 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 22:26:44.351164 16739 webserver.cc:460] Webserver started at http://127.29.106.3:45067/ using document root <none> and password file <none>
I20260811 22:26:44.351840 16739 fs_manager.cc:359] Metadata directory not provided
I20260811 22:26:44.351943 16739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:26:44.352246 16739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:26:44.354959 16739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/data/instance:
uuid: "288ac41534e244f1b3251046bd981d11"
format_stamp: "Formatted at 2026-08-11 22:26:44 on dist-test-slave-gjrz"
server_key: "e1279d8d079a0406382ae0b615f853cf"
server_key_iv: "34bbe773f59dcfa450854db87238c5a9"
server_key_version: "encryptionkey@0"
I20260811 22:26:44.355749 16739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance:
uuid: "288ac41534e244f1b3251046bd981d11"
format_stamp: "Formatted at 2026-08-11 22:26:44 on dist-test-slave-gjrz"
server_key: "e1279d8d079a0406382ae0b615f853cf"
server_key_iv: "34bbe773f59dcfa450854db87238c5a9"
server_key_version: "encryptionkey@0"
I20260811 22:26:44.360572 16739 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260811 22:26:44.363835 16765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:44.365123 16739 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 22:26:44.365273 16739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/wal
uuid: "288ac41534e244f1b3251046bd981d11"
format_stamp: "Formatted at 2026-08-11 22:26:44 on dist-test-slave-gjrz"
server_key: "e1279d8d079a0406382ae0b615f853cf"
server_key_iv: "34bbe773f59dcfa450854db87238c5a9"
server_key_version: "encryptionkey@0"
I20260811 22:26:44.365464 16739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:26:44.370913 16739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:26:44.371695 16739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:26:44.371939 16739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:26:44.372788 16739 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:26:44.374092 16739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:26:44.374188 16739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:44.374294 16739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:26:44.374365 16739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:44.403493 16739 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.3:46529
I20260811 22:26:44.403548 16971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.3:46529 every 8 connection(s)
I20260811 22:26:44.404829 16739 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
I20260811 22:26:44.411748 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 16739
I20260811 22:26:44.411908 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance
I20260811 22:26:44.412451 30120 external_mini_cluster.cc:1428] Setting key cb0db7a72db02e2c1200ca9c3fd279e5
I20260811 22:26:44.412473 16503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:44.407204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:53165 (local address 127.29.106.62:40113)
0811 22:26:44.407388 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:44.407392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:44.408697 (+ 1305us) server_negotiation.cc:407] Connection header received
0811 22:26:44.410336 (+ 1639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:44.410342 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:44.410405 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:44.410492 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:44.411814 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:44.411819 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:44.411823 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:26:44.411845 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:44.411890 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:26:44.411907 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:44.412054 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:44.412218 (+ 164us) server_negotiation.cc:299] Negotiation successful
0811 22:26:44.412271 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260811 22:26:44.413486 16977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:44.407672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:40113 (local address 127.29.106.3:53165)
0811 22:26:44.408546 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:44.408587 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:44.410077 (+ 1490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:44.410714 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:44.410721 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:44.410808 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:44.411467 (+ 659us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 22:26:44.411471 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:44.411521 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 22:26:44.411630 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:26:44.412007 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:26:44.412017 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:26:44.412025 (+ 8us) client_negotiation.cc:769] Sending connection context
0811 22:26:44.412303 (+ 278us) client_negotiation.cc:240] Negotiation successful
0811 22:26:44.412539 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":260,"threads_started":1}
I20260811 22:26:44.414736 16975 heartbeater.cc:347] Connected to a master server at 127.29.106.62:40113
I20260811 22:26:44.415027 16975 heartbeater.cc:464] Registering TS with master...
I20260811 22:26:44.415704 16975 heartbeater.cc:511] Master 127.29.106.62:40113 requested a full tablet report, sending...
I20260811 22:26:44.416893 16163 ts_manager.cc:194] Registered new tserver with Master: 288ac41534e244f1b3251046bd981d11 (127.29.106.3:46529)
I20260811 22:26:44.427412 30120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260811 22:26:44.433257 16503 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:26:44.431779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45006 (local address 127.29.106.62:40113)
0811 22:26:44.431927 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:44.431932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:44.432407 (+ 475us) server_negotiation.cc:407] Connection header received
0811 22:26:44.432560 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:44.432566 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:44.432617 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:44.432694 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:44.433068 (+ 374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45006: BlockingRecv error: recv got EOF from 127.0.0.1:45006 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260811 22:26:44.433943 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 16273
I20260811 22:26:44.437148 16986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:44.434848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50136 (local address 127.29.106.1:37077)
0811 22:26:44.435363 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:44.435369 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:44.435394 (+ 25us) server_negotiation.cc:407] Connection header received
0811 22:26:44.435449 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:44.435454 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:44.435640 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:44.435756 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:44.436566 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:44.436572 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:44.436582 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:26:44.436600 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:44.436638 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:26:44.436651 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:44.436749 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:44.436796 (+ 47us) server_negotiation.cc:299] Negotiation successful
0811 22:26:44.436866 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":223,"threads_started":1}
I20260811 22:26:44.438220 16490 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:26:44.517036 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 16273
I20260811 22:26:44.538177 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 16506
I20260811 22:26:44.541572 16990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:44.539111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44428 (local address 127.29.106.2:42651)
0811 22:26:44.539666 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:44.539673 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:44.539699 (+ 26us) server_negotiation.cc:407] Connection header received
0811 22:26:44.539762 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:44.539769 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:44.539974 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:44.540107 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:44.540893 (+ 786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:44.540900 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:44.540911 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:26:44.540928 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:44.540967 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:26:44.540981 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:44.541094 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:44.541184 (+ 90us) server_negotiation.cc:299] Negotiation successful
0811 22:26:44.541277 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":283,"threads_started":1}
I20260811 22:26:44.542814 16725 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:26:44.622371 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 16506
I20260811 22:26:44.645594 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 16739
I20260811 22:26:44.649551 16995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:44.646504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56314 (local address 127.29.106.3:46529)
0811 22:26:44.647164 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:44.647171 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:44.647203 (+ 32us) server_negotiation.cc:407] Connection header received
0811 22:26:44.647287 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:44.647296 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:44.647526 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:44.647677 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:44.648808 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:44.648814 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:44.648826 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:26:44.648842 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:44.648881 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:26:44.648894 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:44.649065 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:44.649144 (+ 79us) server_negotiation.cc:299] Negotiation successful
0811 22:26:44.649250 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":319,"threads_started":1}
I20260811 22:26:44.650739 16960 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:26:44.730854 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 16739
I20260811 22:26:44.756368 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 16116
I20260811 22:26:44.757591 16224 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:26:44.858982 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 16116
2026-08-11T22:26:44Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2313 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:26:44 dist-test-slave-gjrz krb5kdc[17001](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:26:44 dist-test-slave-gjrz krb5kdc[17001](info): set up 2 sockets
Aug 11 22:26:44 dist-test-slave-gjrz krb5kdc[17001](info): commencing operation
krb5kdc: starting...
W20260811 22:26:46.124115 30120 mini_kdc.cc:121] Time spent starting KDC: real 1.218s user 0.002s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Aug 11 22:26:46 dist-test-slave-gjrz krb5kdc[17001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:26:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:26:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.106.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
I20260811 22:26:46.363490 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:44567
--webserver_interface=127.29.106.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:45929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:44567
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:26:46.686828 17025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:26:46.687095 17025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:26:46.687232 17025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:26:46.694087 17025 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:26:46.694188 17025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:26:46.694268 17025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:26:46.694347 17025 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:26:46.694409 17025 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:26:46.703645 17025 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:45929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:44567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:44567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.106.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:26:46.704789 17025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:26:46.706041 17025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:26:46.717043 17034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:46.717083 17041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:46.717201 17033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:26:46.718307 17025 server_base.cc:1034] running on GCE node
I20260811 22:26:46.719005 17025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:26:46.720295 17025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:26:46.721498 17025 hybrid_clock.cc:648] HybridClock initialized: now 1786487206721453 us; error 33 us; skew 500 ppm
I20260811 22:26:46.721805 17025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:26:46 dist-test-slave-gjrz krb5kdc[17001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487206, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:26:46.724749 17025 init.cc:377] Logged in from keytab as kudu/127.29.106.62@KRBTEST.COM (short username kudu)
I20260811 22:26:46.726079 17025 webserver.cc:460] Webserver started at http://127.29.106.62:41515/ using document root <none> and password file <none>
I20260811 22:26:46.726696 17025 fs_manager.cc:359] Metadata directory not provided
I20260811 22:26:46.726814 17025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:26:46.727099 17025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:26:46.729938 17025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "b5677bb02ac14011a8a823643c12ed46"
format_stamp: "Formatted at 2026-08-11 22:26:46 on dist-test-slave-gjrz"
server_key: "85cd49bcc512bc16b2348a49210bfbcd"
server_key_iv: "c9dbeb3aa4566deab665d2bd405fa872"
server_key_version: "encryptionkey@0"
I20260811 22:26:46.730679 17025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "b5677bb02ac14011a8a823643c12ed46"
format_stamp: "Formatted at 2026-08-11 22:26:46 on dist-test-slave-gjrz"
server_key: "85cd49bcc512bc16b2348a49210bfbcd"
server_key_iv: "c9dbeb3aa4566deab665d2bd405fa872"
server_key_version: "encryptionkey@0"
I20260811 22:26:46.735504 17025 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.001s
I20260811 22:26:46.738680 17055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:46.739923 17025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:26:46.740108 17025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "b5677bb02ac14011a8a823643c12ed46"
format_stamp: "Formatted at 2026-08-11 22:26:46 on dist-test-slave-gjrz"
server_key: "85cd49bcc512bc16b2348a49210bfbcd"
server_key_iv: "c9dbeb3aa4566deab665d2bd405fa872"
server_key_version: "encryptionkey@0"
I20260811 22:26:46.740275 17025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:26:46.755897 17025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:26:46.759603 17025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:26:46.759850 17025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:26:46.776763 17025 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.62:44567
I20260811 22:26:46.776796 17138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.62:44567 every 8 connection(s)
I20260811 22:26:46.777985 17025 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:26:46.781477 17142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:26:46.783701 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 17025
I20260811 22:26:46.783917 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:26:46.784741 30120 external_mini_cluster.cc:1428] Setting key afe76396ef38963c981ea0630b21d1e7
I20260811 22:26:46.793434 17142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5677bb02ac14011a8a823643c12ed46: Bootstrap starting.
I20260811 22:26:46.797049 17142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5677bb02ac14011a8a823643c12ed46: Neither blocks nor log segments found. Creating new log.
I20260811 22:26:46.798318 17142 log.cc:824] T 00000000000000000000000000000000 P b5677bb02ac14011a8a823643c12ed46: Log is configured to *not* fsync() on all Append() calls
Aug 11 22:26:46 dist-test-slave-gjrz krb5kdc[17001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:26:46.801218 17142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5677bb02ac14011a8a823643c12ed46: No bootstrap required, opened a new log
I20260811 22:26:46.807175 17147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:46.786758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56084 (local address 127.29.106.62:44567)
0811 22:26:46.787587 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:46.787599 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:46.787638 (+ 39us) server_negotiation.cc:407] Connection header received
0811 22:26:46.789287 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:46.789308 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:46.789662 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:46.789988 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:46.791783 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:46.793091 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:46.794840 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:46.795577 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:46.795702 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:46.800293 (+ 4591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:46.800307 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:46.800321 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:46.800358 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:46.803339 (+ 2981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:46.803978 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:46.803984 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:46.803990 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:46.804157 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:46.804535 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:46.804539 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:46.804541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:46.805058 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:46.805395 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:46.805816 (+ 421us) server_negotiation.cc:299] Negotiation successful
0811 22:26:46.806216 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":420,"threads_started":1}
I20260811 22:26:46.809500 17142 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5677bb02ac14011a8a823643c12ed46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5677bb02ac14011a8a823643c12ed46" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 44567 } }
I20260811 22:26:46.809864 17142 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5677bb02ac14011a8a823643c12ed46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:26:46.809965 17142 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5677bb02ac14011a8a823643c12ed46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5677bb02ac14011a8a823643c12ed46, State: Initialized, Role: FOLLOWER
I20260811 22:26:46.810527 17142 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5677bb02ac14011a8a823643c12ed46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5677bb02ac14011a8a823643c12ed46" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 44567 } }
I20260811 22:26:46.810739 17142 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5677bb02ac14011a8a823643c12ed46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:26:46.810861 17142 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5677bb02ac14011a8a823643c12ed46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:26:46.811008 17142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5677bb02ac14011a8a823643c12ed46 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:26:46.813159 17142 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5677bb02ac14011a8a823643c12ed46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5677bb02ac14011a8a823643c12ed46" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 44567 } }
I20260811 22:26:46.813611 17142 leader_election.cc:302] T 00000000000000000000000000000000 P b5677bb02ac14011a8a823643c12ed46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5677bb02ac14011a8a823643c12ed46; no voters:
I20260811 22:26:46.814126 17142 leader_election.cc:288] T 00000000000000000000000000000000 P b5677bb02ac14011a8a823643c12ed46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:26:46.814236 17149 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5677bb02ac14011a8a823643c12ed46 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:26:46.815304 17149 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5677bb02ac14011a8a823643c12ed46 [term 1 LEADER]: Becoming Leader. State: Replica: b5677bb02ac14011a8a823643c12ed46, State: Running, Role: LEADER
I20260811 22:26:46.815851 17149 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5677bb02ac14011a8a823643c12ed46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5677bb02ac14011a8a823643c12ed46" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 44567 } }
I20260811 22:26:46.816092 17142 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5677bb02ac14011a8a823643c12ed46 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:26:46.818528 17152 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5677bb02ac14011a8a823643c12ed46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5677bb02ac14011a8a823643c12ed46. Latest consensus state: current_term: 1 leader_uuid: "b5677bb02ac14011a8a823643c12ed46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5677bb02ac14011a8a823643c12ed46" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 44567 } } }
I20260811 22:26:46.818519 17150 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5677bb02ac14011a8a823643c12ed46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5677bb02ac14011a8a823643c12ed46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5677bb02ac14011a8a823643c12ed46" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 44567 } } }
I20260811 22:26:46.818840 17152 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5677bb02ac14011a8a823643c12ed46 [sys.catalog]: This master's current role is: LEADER
I20260811 22:26:46.818892 17150 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5677bb02ac14011a8a823643c12ed46 [sys.catalog]: This master's current role is: LEADER
I20260811 22:26:46.822712 17160 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:26:46.826834 17160 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:26:46.834463 17160 catalog_manager.cc:1347] Generated new cluster ID: 327b5cea4f1e4403aece3319b8850977
I20260811 22:26:46.834563 17160 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:26:46.850811 17160 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:26:46.852545 17160 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:26:46.859637 17160 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5677bb02ac14011a8a823643c12ed46: Generated new TSK 0
I20260811 22:26:46.860337 17160 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.106.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab.
Entry for principal kudu/127.29.106.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab.
I20260811 22:26:46.932606 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.1:0
--local_ip_for_outbound_sockets=127.29.106.1
--webserver_interface=127.29.106.1
--webserver_port=0
--tserver_master_addrs=127.29.106.62:44567
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab
--principal=kudu/127.29.106.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:45929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:26:47.253444 17181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:26:47.253722 17181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:26:47.253856 17181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:26:47.260875 17181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:26:47.261013 17181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:26:47.261158 17181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.1
I20260811 22:26:47.270473 17181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:45929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab
--principal=kudu/127.29.106.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.106.1
--webserver_port=0
--tserver_master_addrs=127.29.106.62:44567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.1
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:26:47.271600 17181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:26:47.272843 17181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:26:47.284818 17191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:47.284940 17196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:47.284969 17192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:26:47.285864 17181 server_base.cc:1034] running on GCE node
I20260811 22:26:47.286434 17181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:26:47.287312 17181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:26:47.288509 17181 hybrid_clock.cc:648] HybridClock initialized: now 1786487207288450 us; error 53 us; skew 500 ppm
I20260811 22:26:47.288789 17181 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:26:47 dist-test-slave-gjrz krb5kdc[17001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487207, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:26:47.291743 17181 init.cc:377] Logged in from keytab as kudu/127.29.106.1@KRBTEST.COM (short username kudu)
I20260811 22:26:47.293040 17181 webserver.cc:460] Webserver started at http://127.29.106.1:40139/ using document root <none> and password file <none>
I20260811 22:26:47.293673 17181 fs_manager.cc:359] Metadata directory not provided
I20260811 22:26:47.293788 17181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:26:47.294066 17181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:26:47.296828 17181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/data/instance:
uuid: "705e0072af094f2eab53a2e55a77be5f"
format_stamp: "Formatted at 2026-08-11 22:26:47 on dist-test-slave-gjrz"
server_key: "89f5898b0cd77fe66739203fb0b888f4"
server_key_iv: "15444cfa053e4e9ffb3bd722868dff0a"
server_key_version: "encryptionkey@0"
I20260811 22:26:47.297560 17181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance:
uuid: "705e0072af094f2eab53a2e55a77be5f"
format_stamp: "Formatted at 2026-08-11 22:26:47 on dist-test-slave-gjrz"
server_key: "89f5898b0cd77fe66739203fb0b888f4"
server_key_iv: "15444cfa053e4e9ffb3bd722868dff0a"
server_key_version: "encryptionkey@0"
I20260811 22:26:47.302311 17181 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260811 22:26:47.305630 17206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:47.307802 17181 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260811 22:26:47.307952 17181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/wal
uuid: "705e0072af094f2eab53a2e55a77be5f"
format_stamp: "Formatted at 2026-08-11 22:26:47 on dist-test-slave-gjrz"
server_key: "89f5898b0cd77fe66739203fb0b888f4"
server_key_iv: "15444cfa053e4e9ffb3bd722868dff0a"
server_key_version: "encryptionkey@0"
I20260811 22:26:47.308127 17181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:26:47.331836 17181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:26:47.335259 17181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:26:47.335564 17181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:26:47.336397 17181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:26:47.337610 17181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:26:47.337687 17181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:47.337800 17181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:26:47.337852 17181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:47.367381 17181 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.1:42915
I20260811 22:26:47.367424 17385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.1:42915 every 8 connection(s)
I20260811 22:26:47.368570 17181 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
I20260811 22:26:47.376771 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 17181
I20260811 22:26:47.376950 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance
I20260811 22:26:47.377543 30120 external_mini_cluster.cc:1428] Setting key a3dfa3a126fd55cc4d130a159a92a2de
Aug 11 22:26:47 dist-test-slave-gjrz krb5kdc[17001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487207, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.1@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:26:47.387449 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:47.370830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:40347 (local address 127.29.106.62:44567)
0811 22:26:47.371217 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:47.371221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:47.372143 (+ 922us) server_negotiation.cc:407] Connection header received
0811 22:26:47.373903 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:47.373909 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:47.373984 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:47.374077 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:47.375508 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:47.376233 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:47.377801 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:47.378187 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:47.378303 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:47.381786 (+ 3483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:47.381791 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:47.381796 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:47.381830 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:47.384691 (+ 2861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:47.385335 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:47.385340 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:47.385343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:47.385481 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:47.385857 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:47.385861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:47.385863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:47.386172 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:47.386303 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:47.386893 (+ 590us) server_negotiation.cc:299] Negotiation successful
0811 22:26:47.387107 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":209,"threads_started":1}
I20260811 22:26:47.388109 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:47.371306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:44567 (local address 127.29.106.1:40347)
0811 22:26:47.371987 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:47.372034 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:47.373643 (+ 1609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:47.374306 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:47.374313 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:47.374778 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:47.375350 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:47.375371 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:47.376397 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:47.376403 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:47.377657 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:47.377672 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:47.378393 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:47.378399 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:47.378674 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:47.379340 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:47.379369 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:47.381574 (+ 2205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:26:47.384816 (+ 3242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:47.384823 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:47.384833 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:47.385216 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:47.385584 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:47.385587 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:47.385589 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:47.385752 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:47.386274 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:26:47.386279 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:26:47.386535 (+ 256us) client_negotiation.cc:769] Sending connection context
0811 22:26:47.386751 (+ 216us) client_negotiation.cc:240] Negotiation successful
0811 22:26:47.387089 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":258,"threads_started":1}
I20260811 22:26:47.389402 17386 heartbeater.cc:347] Connected to a master server at 127.29.106.62:44567
I20260811 22:26:47.389694 17386 heartbeater.cc:464] Registering TS with master...
I20260811 22:26:47.390460 17386 heartbeater.cc:511] Master 127.29.106.62:44567 requested a full tablet report, sending...
I20260811 22:26:47.392535 17079 ts_manager.cc:194] Registered new tserver with Master: 705e0072af094f2eab53a2e55a77be5f (127.29.106.1:42915)
I20260811 22:26:47.393934 17079 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.1@KRBTEST.COM'} at 127.29.106.1:40347
WARNING: no policy specified for kudu/127.29.106.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab.
Entry for principal kudu/127.29.106.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab.
I20260811 22:26:47.431937 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.2:0
--local_ip_for_outbound_sockets=127.29.106.2
--webserver_interface=127.29.106.2
--webserver_port=0
--tserver_master_addrs=127.29.106.62:44567
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab
--principal=kudu/127.29.106.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:45929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:26:47.748658 17397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:26:47.748919 17397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:26:47.749068 17397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:26:47.755941 17397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:26:47.756045 17397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:26:47.756199 17397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.2
I20260811 22:26:47.765420 17397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:45929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab
--principal=kudu/127.29.106.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.106.2
--webserver_port=0
--tserver_master_addrs=127.29.106.62:44567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.2
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:26:47.766523 17397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:26:47.767759 17397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:26:47.780644 17406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:47.780714 17403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:47.781545 17410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:26:47.781858 17397 server_base.cc:1034] running on GCE node
I20260811 22:26:47.782521 17397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:26:47.783526 17397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:26:47.784765 17397 hybrid_clock.cc:648] HybridClock initialized: now 1786487207784683 us; error 68 us; skew 500 ppm
I20260811 22:26:47.785112 17397 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:26:47 dist-test-slave-gjrz krb5kdc[17001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487207, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:26:47.788502 17397 init.cc:377] Logged in from keytab as kudu/127.29.106.2@KRBTEST.COM (short username kudu)
I20260811 22:26:47.793728 17397 webserver.cc:460] Webserver started at http://127.29.106.2:36969/ using document root <none> and password file <none>
I20260811 22:26:47.794330 17397 fs_manager.cc:359] Metadata directory not provided
I20260811 22:26:47.794436 17397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:26:47.794713 17397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:26:47.797379 17397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/data/instance:
uuid: "f49b0cd67c574a95aab9ad6aa430fda3"
format_stamp: "Formatted at 2026-08-11 22:26:47 on dist-test-slave-gjrz"
server_key: "ac9416bb9f440d562a581981f3254d80"
server_key_iv: "42304cffaa0bf99cd893839a89e94aa0"
server_key_version: "encryptionkey@0"
I20260811 22:26:47.798072 17397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance:
uuid: "f49b0cd67c574a95aab9ad6aa430fda3"
format_stamp: "Formatted at 2026-08-11 22:26:47 on dist-test-slave-gjrz"
server_key: "ac9416bb9f440d562a581981f3254d80"
server_key_iv: "42304cffaa0bf99cd893839a89e94aa0"
server_key_version: "encryptionkey@0"
I20260811 22:26:47.802711 17397 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260811 22:26:47.805938 17426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:47.807116 17397 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 22:26:47.807304 17397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/wal
uuid: "f49b0cd67c574a95aab9ad6aa430fda3"
format_stamp: "Formatted at 2026-08-11 22:26:47 on dist-test-slave-gjrz"
server_key: "ac9416bb9f440d562a581981f3254d80"
server_key_iv: "42304cffaa0bf99cd893839a89e94aa0"
server_key_version: "encryptionkey@0"
I20260811 22:26:47.807475 17397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:26:47.823239 17397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:26:47.826803 17397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:26:47.827104 17397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:26:47.828037 17397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:26:47.829231 17397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:26:47.829306 17397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:47.829430 17397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:26:47.829483 17397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:47.858623 17397 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.2:34011
I20260811 22:26:47.858681 17634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.2:34011 every 8 connection(s)
I20260811 22:26:47.859879 17397 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
I20260811 22:26:47.863920 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 17397
I20260811 22:26:47.864071 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance
I20260811 22:26:47.864508 30120 external_mini_cluster.cc:1428] Setting key 86be3c91b56e277c007233abd90f67aa
Aug 11 22:26:47 dist-test-slave-gjrz krb5kdc[17001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487207, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.2@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:26:47.878765 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:47.862364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:41431 (local address 127.29.106.62:44567)
0811 22:26:47.862557 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:47.862561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:47.863746 (+ 1185us) server_negotiation.cc:407] Connection header received
0811 22:26:47.865671 (+ 1925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:47.865677 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:47.865755 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:47.865843 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:47.867463 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:47.868167 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:47.869663 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:47.870125 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:47.870236 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:47.873479 (+ 3243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:47.873484 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:47.873487 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:47.873518 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:47.875933 (+ 2415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:47.876611 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:47.876616 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:47.876618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:47.876742 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:47.877153 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:47.877157 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:47.877159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:47.877423 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:47.877558 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:47.878196 (+ 638us) server_negotiation.cc:299] Negotiation successful
0811 22:26:47.878421 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260811 22:26:47.879664 17642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:47.862821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:44567 (local address 127.29.106.2:41431)
0811 22:26:47.863595 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:47.863636 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:47.865406 (+ 1770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:47.866081 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:47.866090 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:47.866688 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:47.867295 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:47.867322 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:47.868316 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:47.868320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:47.869523 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:47.869536 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:47.870345 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:47.870353 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:47.870579 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:47.871190 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:47.871212 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:47.873267 (+ 2055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:26:47.876073 (+ 2806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:47.876081 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:47.876096 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:47.876490 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:47.876844 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:47.876849 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:47.876853 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:47.877038 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:47.877547 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:26:47.877554 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:26:47.877849 (+ 295us) client_negotiation.cc:769] Sending connection context
0811 22:26:47.878070 (+ 221us) client_negotiation.cc:240] Negotiation successful
0811 22:26:47.878558 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":263,"threads_started":1}
I20260811 22:26:47.880923 17636 heartbeater.cc:347] Connected to a master server at 127.29.106.62:44567
I20260811 22:26:47.881183 17636 heartbeater.cc:464] Registering TS with master...
I20260811 22:26:47.881860 17636 heartbeater.cc:511] Master 127.29.106.62:44567 requested a full tablet report, sending...
I20260811 22:26:47.883165 17079 ts_manager.cc:194] Registered new tserver with Master: f49b0cd67c574a95aab9ad6aa430fda3 (127.29.106.2:34011)
I20260811 22:26:47.883968 17079 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.2@KRBTEST.COM'} at 127.29.106.2:41431
WARNING: no policy specified for kudu/127.29.106.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab.
Entry for principal kudu/127.29.106.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab.
I20260811 22:26:47.920549 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.3:0
--local_ip_for_outbound_sockets=127.29.106.3
--webserver_interface=127.29.106.3
--webserver_port=0
--tserver_master_addrs=127.29.106.62:44567
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab
--principal=kudu/127.29.106.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:45929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:26:48.238202 17645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:26:48.238443 17645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:26:48.238533 17645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:26:48.245476 17645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:26:48.245568 17645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:26:48.245677 17645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.3
I20260811 22:26:48.254706 17645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:45929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab
--principal=kudu/127.29.106.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.106.3
--webserver_port=0
--tserver_master_addrs=127.29.106.62:44567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.3
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:26:48.255879 17645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:26:48.257092 17645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:26:48.269059 17653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:48.268992 17654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:26:48.269070 17657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:26:48.270239 17645 server_base.cc:1034] running on GCE node
I20260811 22:26:48.270725 17645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:26:48.271584 17645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:26:48.272773 17645 hybrid_clock.cc:648] HybridClock initialized: now 1786487208272731 us; error 34 us; skew 500 ppm
I20260811 22:26:48.273067 17645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:26:48 dist-test-slave-gjrz krb5kdc[17001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487208, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:26:48.275924 17645 init.cc:377] Logged in from keytab as kudu/127.29.106.3@KRBTEST.COM (short username kudu)
I20260811 22:26:48.277216 17645 webserver.cc:460] Webserver started at http://127.29.106.3:33129/ using document root <none> and password file <none>
I20260811 22:26:48.277813 17645 fs_manager.cc:359] Metadata directory not provided
I20260811 22:26:48.277927 17645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:26:48.278198 17645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:26:48.280910 17645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/data/instance:
uuid: "97edea8ccfb74184b1576d4868ff608a"
format_stamp: "Formatted at 2026-08-11 22:26:48 on dist-test-slave-gjrz"
server_key: "fcf89e93316174c15033566a6aab3c62"
server_key_iv: "9451ed0400abdb4d11159411760d3536"
server_key_version: "encryptionkey@0"
I20260811 22:26:48.281668 17645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance:
uuid: "97edea8ccfb74184b1576d4868ff608a"
format_stamp: "Formatted at 2026-08-11 22:26:48 on dist-test-slave-gjrz"
server_key: "fcf89e93316174c15033566a6aab3c62"
server_key_iv: "9451ed0400abdb4d11159411760d3536"
server_key_version: "encryptionkey@0"
I20260811 22:26:48.286363 17645 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260811 22:26:48.289441 17680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:48.290665 17645 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 22:26:48.290815 17645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/wal
uuid: "97edea8ccfb74184b1576d4868ff608a"
format_stamp: "Formatted at 2026-08-11 22:26:48 on dist-test-slave-gjrz"
server_key: "fcf89e93316174c15033566a6aab3c62"
server_key_iv: "9451ed0400abdb4d11159411760d3536"
server_key_version: "encryptionkey@0"
I20260811 22:26:48.291028 17645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:26:48.307837 17645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:26:48.311470 17645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:26:48.311750 17645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:26:48.312626 17645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:26:48.313978 17645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:26:48.314061 17645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:48.314179 17645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:26:48.314234 17645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:26:48.344483 17645 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.3:36021
I20260811 22:26:48.344573 17881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.3:36021 every 8 connection(s)
I20260811 22:26:48.345685 17645 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
I20260811 22:26:48.353843 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 17645
I20260811 22:26:48.353989 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance
I20260811 22:26:48.354410 30120 external_mini_cluster.cc:1428] Setting key d6d2b4b91b4b5eeb7a197c4040811648
Aug 11 22:26:48 dist-test-slave-gjrz krb5kdc[17001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487208, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.3@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:26:48.364420 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:48.348043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:46453 (local address 127.29.106.62:44567)
0811 22:26:48.348218 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:48.348224 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:48.349297 (+ 1073us) server_negotiation.cc:407] Connection header received
0811 22:26:48.351071 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:48.351077 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:48.351180 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:48.351287 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:48.352711 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:48.353469 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:48.355161 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:48.355766 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:48.355897 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:48.359345 (+ 3448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:48.359350 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:48.359354 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:48.359386 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:48.361976 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:48.362600 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:48.362603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:48.362605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:48.362725 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:48.363048 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:48.363051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:48.363053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:48.363341 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:48.363454 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:48.364022 (+ 568us) server_negotiation.cc:299] Negotiation successful
0811 22:26:48.364183 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 22:26:48.365257 17887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:48.348462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:44567 (local address 127.29.106.3:46453)
0811 22:26:48.349148 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:48.349188 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:48.350799 (+ 1611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:48.351528 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:48.351536 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:48.352015 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:26:48.352540 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:48.352568 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:48.353640 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:48.353644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:48.354989 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:48.355005 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:48.355963 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:48.355968 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:48.356229 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:48.356872 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:26:48.356901 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:26:48.359143 (+ 2242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:26:48.362096 (+ 2953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:48.362102 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:48.362114 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:48.362498 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:48.362799 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:26:48.362802 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:26:48.362804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:26:48.362974 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:26:48.363441 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:26:48.363446 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:26:48.363740 (+ 294us) client_negotiation.cc:769] Sending connection context
0811 22:26:48.363953 (+ 213us) client_negotiation.cc:240] Negotiation successful
0811 22:26:48.364289 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":271,"threads_started":1}
I20260811 22:26:48.366457 17882 heartbeater.cc:347] Connected to a master server at 127.29.106.62:44567
I20260811 22:26:48.366726 17882 heartbeater.cc:464] Registering TS with master...
I20260811 22:26:48.367578 17882 heartbeater.cc:511] Master 127.29.106.62:44567 requested a full tablet report, sending...
I20260811 22:26:48.369354 17078 ts_manager.cc:194] Registered new tserver with Master: 97edea8ccfb74184b1576d4868ff608a (127.29.106.3:36021)
I20260811 22:26:48.370539 17078 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.3@KRBTEST.COM'} at 127.29.106.3:46453
I20260811 22:26:48.371376 30120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 22:26:48.387008 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:48.375621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56100 (local address 127.29.106.62:44567)
0811 22:26:48.375794 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:48.375800 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:48.376182 (+ 382us) server_negotiation.cc:407] Connection header received
0811 22:26:48.376296 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:48.376299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:48.376354 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:48.376427 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:48.377715 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:48.378437 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:48.379611 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:48.380087 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:48.380180 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:48.382207 (+ 2027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:48.382212 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:48.382215 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:48.382242 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:48.384558 (+ 2316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:48.385133 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:48.385136 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:48.385139 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:48.385249 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:48.385534 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:48.385536 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:48.385539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:48.385884 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:48.386119 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:48.386493 (+ 374us) server_negotiation.cc:299] Negotiation successful
0811 22:26:48.386698 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260811 22:26:48.390620 17079 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56100:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260811 22:26:48.396670 17386 heartbeater.cc:503] Master 127.29.106.62:44567 was elected leader, sending a full tablet report...
W20260811 22:26:48.397637 17079 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:26:48.421157 17905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:48.408744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.2:34011 (local address 127.0.0.1:56214)
0811 22:26:48.409531 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:48.409585 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:48.409760 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:48.410673 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:48.410681 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:48.410708 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:26:48.411124 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:48.411138 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:48.414556 (+ 3418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:48.414562 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:48.417084 (+ 2522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:48.417101 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:48.420168 (+ 3067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:48.420175 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:48.420405 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:48.420460 (+ 55us) client_negotiation.cc:769] Sending connection context
0811 22:26:48.420567 (+ 107us) client_negotiation.cc:240] Negotiation successful
0811 22:26:48.420769 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":409,"threads_started":1}
I20260811 22:26:48.424170 17906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:48.408951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56214 (local address 127.29.106.2:34011)
0811 22:26:48.410061 (+ 1110us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:48.410069 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:48.410094 (+ 25us) server_negotiation.cc:407] Connection header received
0811 22:26:48.410188 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:48.410196 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:48.410427 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:48.411391 (+ 964us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:26:48.411690 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:48.414382 (+ 2692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:48.418316 (+ 3934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:48.419916 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:48.423388 (+ 3472us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:48.423605 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:48.423683 (+ 78us) server_negotiation.cc:299] Negotiation successful
0811 22:26:48.423853 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":961,"thread_start_us":839,"threads_started":1}
I20260811 22:26:48.429234 17905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:48.410616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.3:36021 (local address 127.0.0.1:46446)
0811 22:26:48.421422 (+ 10806us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:48.421441 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:48.421541 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:48.422639 (+ 1098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:48.422644 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:48.422674 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:26:48.422863 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:48.422878 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:48.425028 (+ 2150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:48.425034 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:48.426688 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:48.426703 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:48.428411 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:48.428417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:48.428619 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:48.428658 (+ 39us) client_negotiation.cc:769] Sending connection context
0811 22:26:48.428748 (+ 90us) client_negotiation.cc:240] Negotiation successful
0811 22:26:48.428892 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10701,"thread_start_us":14066,"threads_started":1}
I20260811 22:26:48.429291 17910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:48.411177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46446 (local address 127.29.106.3:36021)
0811 22:26:48.421854 (+ 10677us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:48.421862 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:48.421916 (+ 54us) server_negotiation.cc:407] Connection header received
0811 22:26:48.422014 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:48.422021 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:48.422262 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:48.423041 (+ 779us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:26:48.423349 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:48.424853 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:48.426857 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:48.428234 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:48.428361 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:48.428585 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:48.428810 (+ 225us) server_negotiation.cc:299] Negotiation successful
0811 22:26:48.428964 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10507,"thread_start_us":11194,"threads_started":1}
I20260811 22:26:48.428737 17513 tablet_service.cc:1467] Processing CreateTablet for tablet 3fc200fd6e184514b42f6e41df517922 (DEFAULT_TABLE table=test-table [id=78080626114747e185e6d13af79adb49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:26:48.430296 17513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fc200fd6e184514b42f6e41df517922. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:26:48.432417 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:48.409996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:42915 (local address 127.0.0.1:55810)
0811 22:26:48.410501 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:48.410519 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:48.410616 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:48.425916 (+ 15300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:48.425921 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:48.425953 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:26:48.426398 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:48.426411 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:48.428118 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:48.428123 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:48.429949 (+ 1826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:48.429961 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:48.431676 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:48.431683 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:48.431848 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:48.431878 (+ 30us) client_negotiation.cc:769] Sending connection context
0811 22:26:48.431982 (+ 104us) client_negotiation.cc:240] Negotiation successful
0811 22:26:48.432105 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":250,"threads_started":1}
I20260811 22:26:48.432510 17908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:48.410250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55810 (local address 127.29.106.1:42915)
0811 22:26:48.425315 (+ 15065us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:48.425324 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:48.425350 (+ 26us) server_negotiation.cc:407] Connection header received
0811 22:26:48.425443 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:48.425451 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:48.425675 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:48.425905 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:26:48.426608 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:48.427972 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:48.430095 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:48.431456 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:48.431592 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:48.431800 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:48.432036 (+ 236us) server_negotiation.cc:299] Negotiation successful
0811 22:26:48.432212 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14897,"thread_start_us":14720,"threads_started":1}
I20260811 22:26:48.432906 17756 tablet_service.cc:1467] Processing CreateTablet for tablet 3fc200fd6e184514b42f6e41df517922 (DEFAULT_TABLE table=test-table [id=78080626114747e185e6d13af79adb49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:26:48.434180 17756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fc200fd6e184514b42f6e41df517922. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:26:48.441646 17911 tablet_bootstrap.cc:492] T 3fc200fd6e184514b42f6e41df517922 P f49b0cd67c574a95aab9ad6aa430fda3: Bootstrap starting.
I20260811 22:26:48.443310 17282 tablet_service.cc:1467] Processing CreateTablet for tablet 3fc200fd6e184514b42f6e41df517922 (DEFAULT_TABLE table=test-table [id=78080626114747e185e6d13af79adb49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:26:48.444628 17282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fc200fd6e184514b42f6e41df517922. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:26:48.444859 17911 tablet_bootstrap.cc:654] T 3fc200fd6e184514b42f6e41df517922 P f49b0cd67c574a95aab9ad6aa430fda3: Neither blocks nor log segments found. Creating new log.
I20260811 22:26:48.445978 17911 log.cc:824] T 3fc200fd6e184514b42f6e41df517922 P f49b0cd67c574a95aab9ad6aa430fda3: Log is configured to *not* fsync() on all Append() calls
I20260811 22:26:48.446041 17912 tablet_bootstrap.cc:492] T 3fc200fd6e184514b42f6e41df517922 P 97edea8ccfb74184b1576d4868ff608a: Bootstrap starting.
I20260811 22:26:48.449357 17911 tablet_bootstrap.cc:492] T 3fc200fd6e184514b42f6e41df517922 P f49b0cd67c574a95aab9ad6aa430fda3: No bootstrap required, opened a new log
I20260811 22:26:48.449350 17912 tablet_bootstrap.cc:654] T 3fc200fd6e184514b42f6e41df517922 P 97edea8ccfb74184b1576d4868ff608a: Neither blocks nor log segments found. Creating new log.
I20260811 22:26:48.449676 17911 ts_tablet_manager.cc:1397] T 3fc200fd6e184514b42f6e41df517922 P f49b0cd67c574a95aab9ad6aa430fda3: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260811 22:26:48.450580 17912 log.cc:824] T 3fc200fd6e184514b42f6e41df517922 P 97edea8ccfb74184b1576d4868ff608a: Log is configured to *not* fsync() on all Append() calls
I20260811 22:26:48.453482 17912 tablet_bootstrap.cc:492] T 3fc200fd6e184514b42f6e41df517922 P 97edea8ccfb74184b1576d4868ff608a: No bootstrap required, opened a new log
I20260811 22:26:48.453740 17912 ts_tablet_manager.cc:1397] T 3fc200fd6e184514b42f6e41df517922 P 97edea8ccfb74184b1576d4868ff608a: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260811 22:26:48.455597 17920 tablet_bootstrap.cc:492] T 3fc200fd6e184514b42f6e41df517922 P 705e0072af094f2eab53a2e55a77be5f: Bootstrap starting.
I20260811 22:26:48.458191 17920 tablet_bootstrap.cc:654] T 3fc200fd6e184514b42f6e41df517922 P 705e0072af094f2eab53a2e55a77be5f: Neither blocks nor log segments found. Creating new log.
I20260811 22:26:48.459183 17920 log.cc:824] T 3fc200fd6e184514b42f6e41df517922 P 705e0072af094f2eab53a2e55a77be5f: Log is configured to *not* fsync() on all Append() calls
I20260811 22:26:48.459635 17911 raft_consensus.cc:348] T 3fc200fd6e184514b42f6e41df517922 P f49b0cd67c574a95aab9ad6aa430fda3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49b0cd67c574a95aab9ad6aa430fda3" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 34011 } } peers { permanent_uuid: "705e0072af094f2eab53a2e55a77be5f" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 42915 } } peers { permanent_uuid: "97edea8ccfb74184b1576d4868ff608a" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36021 } }
I20260811 22:26:48.460161 17911 raft_consensus.cc:374] T 3fc200fd6e184514b42f6e41df517922 P f49b0cd67c574a95aab9ad6aa430fda3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:26:48.460331 17911 raft_consensus.cc:729] T 3fc200fd6e184514b42f6e41df517922 P f49b0cd67c574a95aab9ad6aa430fda3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f49b0cd67c574a95aab9ad6aa430fda3, State: Initialized, Role: FOLLOWER
I20260811 22:26:48.460932 17911 consensus_queue.cc:260] T 3fc200fd6e184514b42f6e41df517922 P f49b0cd67c574a95aab9ad6aa430fda3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49b0cd67c574a95aab9ad6aa430fda3" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 34011 } } peers { permanent_uuid: "705e0072af094f2eab53a2e55a77be5f" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 42915 } } peers { permanent_uuid: "97edea8ccfb74184b1576d4868ff608a" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36021 } }
I20260811 22:26:48.462055 17636 heartbeater.cc:503] Master 127.29.106.62:44567 was elected leader, sending a full tablet report...
I20260811 22:26:48.462312 17920 tablet_bootstrap.cc:492] T 3fc200fd6e184514b42f6e41df517922 P 705e0072af094f2eab53a2e55a77be5f: No bootstrap required, opened a new log
I20260811 22:26:48.462548 17920 ts_tablet_manager.cc:1397] T 3fc200fd6e184514b42f6e41df517922 P 705e0072af094f2eab53a2e55a77be5f: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260811 22:26:48.462635 17911 ts_tablet_manager.cc:1428] T 3fc200fd6e184514b42f6e41df517922 P f49b0cd67c574a95aab9ad6aa430fda3: Time spent starting tablet: real 0.013s user 0.009s sys 0.002s
I20260811 22:26:48.463532 17912 raft_consensus.cc:348] T 3fc200fd6e184514b42f6e41df517922 P 97edea8ccfb74184b1576d4868ff608a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49b0cd67c574a95aab9ad6aa430fda3" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 34011 } } peers { permanent_uuid: "705e0072af094f2eab53a2e55a77be5f" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 42915 } } peers { permanent_uuid: "97edea8ccfb74184b1576d4868ff608a" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36021 } }
I20260811 22:26:48.464073 17912 raft_consensus.cc:374] T 3fc200fd6e184514b42f6e41df517922 P 97edea8ccfb74184b1576d4868ff608a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:26:48.464222 17912 raft_consensus.cc:729] T 3fc200fd6e184514b42f6e41df517922 P 97edea8ccfb74184b1576d4868ff608a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97edea8ccfb74184b1576d4868ff608a, State: Initialized, Role: FOLLOWER
I20260811 22:26:48.464776 17912 consensus_queue.cc:260] T 3fc200fd6e184514b42f6e41df517922 P 97edea8ccfb74184b1576d4868ff608a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49b0cd67c574a95aab9ad6aa430fda3" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 34011 } } peers { permanent_uuid: "705e0072af094f2eab53a2e55a77be5f" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 42915 } } peers { permanent_uuid: "97edea8ccfb74184b1576d4868ff608a" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36021 } }
I20260811 22:26:48.465823 17882 heartbeater.cc:503] Master 127.29.106.62:44567 was elected leader, sending a full tablet report...
I20260811 22:26:48.466478 17912 ts_tablet_manager.cc:1428] T 3fc200fd6e184514b42f6e41df517922 P 97edea8ccfb74184b1576d4868ff608a: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260811 22:26:48.471290 17920 raft_consensus.cc:348] T 3fc200fd6e184514b42f6e41df517922 P 705e0072af094f2eab53a2e55a77be5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49b0cd67c574a95aab9ad6aa430fda3" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 34011 } } peers { permanent_uuid: "705e0072af094f2eab53a2e55a77be5f" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 42915 } } peers { permanent_uuid: "97edea8ccfb74184b1576d4868ff608a" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36021 } }
I20260811 22:26:48.471585 17920 raft_consensus.cc:374] T 3fc200fd6e184514b42f6e41df517922 P 705e0072af094f2eab53a2e55a77be5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:26:48.471658 17920 raft_consensus.cc:729] T 3fc200fd6e184514b42f6e41df517922 P 705e0072af094f2eab53a2e55a77be5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 705e0072af094f2eab53a2e55a77be5f, State: Initialized, Role: FOLLOWER
I20260811 22:26:48.472165 17920 consensus_queue.cc:260] T 3fc200fd6e184514b42f6e41df517922 P 705e0072af094f2eab53a2e55a77be5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49b0cd67c574a95aab9ad6aa430fda3" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 34011 } } peers { permanent_uuid: "705e0072af094f2eab53a2e55a77be5f" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 42915 } } peers { permanent_uuid: "97edea8ccfb74184b1576d4868ff608a" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36021 } }
I20260811 22:26:48.473639 17920 ts_tablet_manager.cc:1428] T 3fc200fd6e184514b42f6e41df517922 P 705e0072af094f2eab53a2e55a77be5f: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
W20260811 22:26:48.597229 17885 tablet.cc:2369] T 3fc200fd6e184514b42f6e41df517922 P 97edea8ccfb74184b1576d4868ff608a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 22:26:48.611717 17641 tablet.cc:2369] T 3fc200fd6e184514b42f6e41df517922 P f49b0cd67c574a95aab9ad6aa430fda3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 22:26:48.620651 17388 tablet.cc:2369] T 3fc200fd6e184514b42f6e41df517922 P 705e0072af094f2eab53a2e55a77be5f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:26:48.623211 17926 raft_consensus.cc:482] T 3fc200fd6e184514b42f6e41df517922 P 705e0072af094f2eab53a2e55a77be5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:26:48.623499 17926 raft_consensus.cc:504] T 3fc200fd6e184514b42f6e41df517922 P 705e0072af094f2eab53a2e55a77be5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49b0cd67c574a95aab9ad6aa430fda3" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 34011 } } peers { permanent_uuid: "705e0072af094f2eab53a2e55a77be5f" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 42915 } } peers { permanent_uuid: "97edea8ccfb74184b1576d4868ff608a" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36021 } }
I20260811 22:26:48.625078 17926 leader_election.cc:288] T 3fc200fd6e184514b42f6e41df517922 P 705e0072af094f2eab53a2e55a77be5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f49b0cd67c574a95aab9ad6aa430fda3 (127.29.106.2:34011), 97edea8ccfb74184b1576d4868ff608a (127.29.106.3:36021)
I20260811 22:26:48.632741 17906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:48.625436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:45489 (local address 127.29.106.2:34011)
0811 22:26:48.625651 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:48.625657 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:48.626061 (+ 404us) server_negotiation.cc:407] Connection header received
0811 22:26:48.626255 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:48.626262 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:48.626341 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:48.626429 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:26:48.627154 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:48.628326 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:48.630854 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:48.631945 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:48.632122 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:48.632195 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:48.632390 (+ 195us) server_negotiation.cc:299] Negotiation successful
0811 22:26:48.632515 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260811 22:26:48.632841 17928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:48.625368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.2:34011 (local address 127.29.106.1:45489)
0811 22:26:48.625946 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:48.625962 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:48.626117 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:48.626522 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:48.626527 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:48.626561 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:26:48.626966 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:48.626980 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:48.628492 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:48.628498 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:48.630695 (+ 2197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:48.630711 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:48.632130 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:48.632134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:48.632252 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:48.632276 (+ 24us) client_negotiation.cc:769] Sending connection context
0811 22:26:48.632350 (+ 74us) client_negotiation.cc:240] Negotiation successful
0811 22:26:48.632434 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":1,"thread_start_us":312,"threads_started":1}
I20260811 22:26:48.633353 17910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:48.626218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:50191 (local address 127.29.106.3:36021)
0811 22:26:48.626425 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:48.626431 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:48.626817 (+ 386us) server_negotiation.cc:407] Connection header received
0811 22:26:48.627085 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:48.627090 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:48.627164 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:48.627248 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:26:48.627990 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:48.629028 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:48.631234 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:48.632398 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:48.632545 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:48.632620 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:48.632998 (+ 378us) server_negotiation.cc:299] Negotiation successful
0811 22:26:48.633123 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260811 22:26:48.633819 17930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:48.626068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.3:36021 (local address 127.29.106.1:50191)
0811 22:26:48.626720 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0811 22:26:48.626734 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 22:26:48.626931 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:26:48.627384 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:26:48.627390 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:26:48.627416 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:26:48.627844 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:48.627858 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:48.629211 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:48.629219 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:48.631095 (+ 1876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:26:48.631106 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:26:48.632624 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:26:48.632630 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:26:48.632802 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:48.632839 (+ 37us) client_negotiation.cc:769] Sending connection context
0811 22:26:48.633488 (+ 649us) client_negotiation.cc:240] Negotiation successful
0811 22:26:48.633605 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":272,"threads_started":1}
I20260811 22:26:48.633909 17545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc200fd6e184514b42f6e41df517922" candidate_uuid: "705e0072af094f2eab53a2e55a77be5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f49b0cd67c574a95aab9ad6aa430fda3" is_pre_election: true
I20260811 22:26:48.634441 17545 raft_consensus.cc:2454] T 3fc200fd6e184514b42f6e41df517922 P f49b0cd67c574a95aab9ad6aa430fda3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 705e0072af094f2eab53a2e55a77be5f in term 0.
I20260811 22:26:48.634735 17801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc200fd6e184514b42f6e41df517922" candidate_uuid: "705e0072af094f2eab53a2e55a77be5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97edea8ccfb74184b1576d4868ff608a" is_pre_election: true
I20260811 22:26:48.635144 17208 leader_election.cc:302] T 3fc200fd6e184514b42f6e41df517922 P 705e0072af094f2eab53a2e55a77be5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 705e0072af094f2eab53a2e55a77be5f, f49b0cd67c574a95aab9ad6aa430fda3; no voters:
I20260811 22:26:48.635183 17801 raft_consensus.cc:2454] T 3fc200fd6e184514b42f6e41df517922 P 97edea8ccfb74184b1576d4868ff608a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 705e0072af094f2eab53a2e55a77be5f in term 0.
I20260811 22:26:48.635516 17926 raft_consensus.cc:2781] T 3fc200fd6e184514b42f6e41df517922 P 705e0072af094f2eab53a2e55a77be5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:26:48.635665 17926 raft_consensus.cc:482] T 3fc200fd6e184514b42f6e41df517922 P 705e0072af094f2eab53a2e55a77be5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:26:48.635778 17926 raft_consensus.cc:3037] T 3fc200fd6e184514b42f6e41df517922 P 705e0072af094f2eab53a2e55a77be5f [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:26:48.637933 17926 raft_consensus.cc:504] T 3fc200fd6e184514b42f6e41df517922 P 705e0072af094f2eab53a2e55a77be5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49b0cd67c574a95aab9ad6aa430fda3" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 34011 } } peers { permanent_uuid: "705e0072af094f2eab53a2e55a77be5f" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 42915 } } peers { permanent_uuid: "97edea8ccfb74184b1576d4868ff608a" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36021 } }
I20260811 22:26:48.638640 17926 leader_election.cc:288] T 3fc200fd6e184514b42f6e41df517922 P 705e0072af094f2eab53a2e55a77be5f [CANDIDATE]: Term 1 election: Requested vote from peers f49b0cd67c574a95aab9ad6aa430fda3 (127.29.106.2:34011), 97edea8ccfb74184b1576d4868ff608a (127.29.106.3:36021)
I20260811 22:26:48.639063 17545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc200fd6e184514b42f6e41df517922" candidate_uuid: "705e0072af094f2eab53a2e55a77be5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f49b0cd67c574a95aab9ad6aa430fda3"
I20260811 22:26:48.639122 17801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc200fd6e184514b42f6e41df517922" candidate_uuid: "705e0072af094f2eab53a2e55a77be5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97edea8ccfb74184b1576d4868ff608a"
I20260811 22:26:48.639228 17545 raft_consensus.cc:3037] T 3fc200fd6e184514b42f6e41df517922 P f49b0cd67c574a95aab9ad6aa430fda3 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:26:48.639345 17801 raft_consensus.cc:3037] T 3fc200fd6e184514b42f6e41df517922 P 97edea8ccfb74184b1576d4868ff608a [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:26:48.641331 17545 raft_consensus.cc:2454] T 3fc200fd6e184514b42f6e41df517922 P f49b0cd67c574a95aab9ad6aa430fda3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 705e0072af094f2eab53a2e55a77be5f in term 1.
I20260811 22:26:48.641436 17801 raft_consensus.cc:2454] T 3fc200fd6e184514b42f6e41df517922 P 97edea8ccfb74184b1576d4868ff608a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 705e0072af094f2eab53a2e55a77be5f in term 1.
I20260811 22:26:48.641666 17208 leader_election.cc:302] T 3fc200fd6e184514b42f6e41df517922 P 705e0072af094f2eab53a2e55a77be5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 705e0072af094f2eab53a2e55a77be5f, f49b0cd67c574a95aab9ad6aa430fda3; no voters:
I20260811 22:26:48.641928 17926 raft_consensus.cc:2781] T 3fc200fd6e184514b42f6e41df517922 P 705e0072af094f2eab53a2e55a77be5f [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:26:48.642414 17926 raft_consensus.cc:686] T 3fc200fd6e184514b42f6e41df517922 P 705e0072af094f2eab53a2e55a77be5f [term 1 LEADER]: Becoming Leader. State: Replica: 705e0072af094f2eab53a2e55a77be5f, State: Running, Role: LEADER
I20260811 22:26:48.642926 17926 consensus_queue.cc:237] T 3fc200fd6e184514b42f6e41df517922 P 705e0072af094f2eab53a2e55a77be5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49b0cd67c574a95aab9ad6aa430fda3" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 34011 } } peers { permanent_uuid: "705e0072af094f2eab53a2e55a77be5f" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 42915 } } peers { permanent_uuid: "97edea8ccfb74184b1576d4868ff608a" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36021 } }
I20260811 22:26:48.647390 17078 catalog_manager.cc:5495] T 3fc200fd6e184514b42f6e41df517922 P 705e0072af094f2eab53a2e55a77be5f reported cstate change: term changed from 0 to 1, leader changed from <none> to 705e0072af094f2eab53a2e55a77be5f (127.29.106.1). New cstate: current_term: 1 leader_uuid: "705e0072af094f2eab53a2e55a77be5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f49b0cd67c574a95aab9ad6aa430fda3" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 34011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "705e0072af094f2eab53a2e55a77be5f" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 42915 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97edea8ccfb74184b1576d4868ff608a" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36021 } health_report { overall_health: UNKNOWN } } }
I20260811 22:26:48.710867 17908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:48.706161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55814 (local address 127.29.106.1:42915)
0811 22:26:48.706333 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:48.706337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:48.706354 (+ 17us) server_negotiation.cc:407] Connection header received
0811 22:26:48.706532 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:48.706536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:48.706626 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:48.706748 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:26:48.707031 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:48.707778 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:48.709025 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:48.709429 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:48.709514 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:48.709908 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:26:48.710379 (+ 471us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:26:48.710497 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:48.710535 (+ 38us) server_negotiation.cc:299] Negotiation successful
0811 22:26:48.710612 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260811 22:26:48.718954 17545 raft_consensus.cc:1261] T 3fc200fd6e184514b42f6e41df517922 P f49b0cd67c574a95aab9ad6aa430fda3 [term 1 FOLLOWER]: Refusing update from remote peer 705e0072af094f2eab53a2e55a77be5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:26:48.718952 17801 raft_consensus.cc:1261] T 3fc200fd6e184514b42f6e41df517922 P 97edea8ccfb74184b1576d4868ff608a [term 1 FOLLOWER]: Refusing update from remote peer 705e0072af094f2eab53a2e55a77be5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:26:48.719921 17931 consensus_queue.cc:1035] T 3fc200fd6e184514b42f6e41df517922 P 705e0072af094f2eab53a2e55a77be5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f49b0cd67c574a95aab9ad6aa430fda3" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 34011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:26:48.720235 17926 consensus_queue.cc:1035] T 3fc200fd6e184514b42f6e41df517922 P 705e0072af094f2eab53a2e55a77be5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "97edea8ccfb74184b1576d4868ff608a" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:26:48.730777 17939 mvcc.cc:205] Tried to move back new op lower bound from 7317451606903070720 to 7317451606605152256. Current Snapshot: MvccSnapshot[applied={T|T < 7317451606903070720}]
I20260811 22:26:48.730849 17937 mvcc.cc:205] Tried to move back new op lower bound from 7317451606903070720 to 7317451606605152256. Current Snapshot: MvccSnapshot[applied={T|T < 7317451606903070720}]
I20260811 22:26:48.743880 17079 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:26:48.744136 17079 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:26:48.747462 17079 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b5677bb02ac14011a8a823643c12ed46: Sending DeleteTablet for 3 replicas of tablet 3fc200fd6e184514b42f6e41df517922
I20260811 22:26:48.748581 17513 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fc200fd6e184514b42f6e41df517922 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:26:48 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:56214
I20260811 22:26:48.748937 17756 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fc200fd6e184514b42f6e41df517922 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:26:48 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:46446
I20260811 22:26:48.749828 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 17181
I20260811 22:26:48.749806 17282 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fc200fd6e184514b42f6e41df517922 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:26:48 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:55810
I20260811 22:26:48.759462 17956 tablet_replica.cc:330] T 3fc200fd6e184514b42f6e41df517922 P 97edea8ccfb74184b1576d4868ff608a: stopping tablet replica
I20260811 22:26:48.760492 17956 raft_consensus.cc:2229] T 3fc200fd6e184514b42f6e41df517922 P 97edea8ccfb74184b1576d4868ff608a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:26:48.761523 17956 raft_consensus.cc:2258] T 3fc200fd6e184514b42f6e41df517922 P 97edea8ccfb74184b1576d4868ff608a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:26:48.764612 17956 ts_tablet_manager.cc:1905] T 3fc200fd6e184514b42f6e41df517922 P 97edea8ccfb74184b1576d4868ff608a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:26:48.767222 17957 tablet_replica.cc:330] T 3fc200fd6e184514b42f6e41df517922 P 705e0072af094f2eab53a2e55a77be5f: stopping tablet replica
I20260811 22:26:48.767671 17957 raft_consensus.cc:2229] T 3fc200fd6e184514b42f6e41df517922 P 705e0072af094f2eab53a2e55a77be5f [term 1 LEADER]: Raft consensus shutting down.
I20260811 22:26:48.768496 17957 raft_consensus.cc:2258] T 3fc200fd6e184514b42f6e41df517922 P 705e0072af094f2eab53a2e55a77be5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:26:48.770380 17957 ts_tablet_manager.cc:1905] T 3fc200fd6e184514b42f6e41df517922 P 705e0072af094f2eab53a2e55a77be5f: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 11 22:26:48 dist-test-slave-gjrz krb5kdc[17001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.1@KRBTEST.COM
I20260811 22:26:48.774075 17956 ts_tablet_manager.cc:1918] T 3fc200fd6e184514b42f6e41df517922 P 97edea8ccfb74184b1576d4868ff608a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:26:48.774245 17956 log.cc:1196] T 3fc200fd6e184514b42f6e41df517922 P 97edea8ccfb74184b1576d4868ff608a: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-2/wal/wals/3fc200fd6e184514b42f6e41df517922
I20260811 22:26:48.774804 17956 ts_tablet_manager.cc:1939] T 3fc200fd6e184514b42f6e41df517922 P 97edea8ccfb74184b1576d4868ff608a: Deleting consensus metadata
I20260811 22:26:48.778002 17957 ts_tablet_manager.cc:1918] T 3fc200fd6e184514b42f6e41df517922 P 705e0072af094f2eab53a2e55a77be5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:26:48.778213 17957 log.cc:1196] T 3fc200fd6e184514b42f6e41df517922 P 705e0072af094f2eab53a2e55a77be5f: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-0/wal/wals/3fc200fd6e184514b42f6e41df517922
I20260811 22:26:48.778744 17957 ts_tablet_manager.cc:1939] T 3fc200fd6e184514b42f6e41df517922 P 705e0072af094f2eab53a2e55a77be5f: Deleting consensus metadata
I20260811 22:26:48.780515 17056 catalog_manager.cc:4841] TS 705e0072af094f2eab53a2e55a77be5f (127.29.106.1:42915): tablet 3fc200fd6e184514b42f6e41df517922 (table test-table [id=78080626114747e185e6d13af79adb49]) successfully deleted
I20260811 22:26:48.781245 17908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:48.751946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55816 (local address 127.29.106.1:42915)
0811 22:26:48.752123 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:48.752130 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:48.761343 (+ 9213us) server_negotiation.cc:407] Connection header received
0811 22:26:48.761785 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:48.761792 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:48.761894 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:48.761971 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:48.763764 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:48.765007 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:48.767434 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:48.768190 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:48.768302 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:48.773637 (+ 5335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:48.773652 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:48.773669 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:48.773720 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:48.777770 (+ 4050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:48.778620 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:48.778628 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:48.778632 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:48.778811 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:48.779314 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:48.779319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:48.779322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:48.779785 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:48.780083 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:48.780669 (+ 586us) server_negotiation.cc:299] Negotiation successful
0811 22:26:48.780896 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 22:26:48.782582 17378 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:26:48.782853 17955 tablet_replica.cc:330] T 3fc200fd6e184514b42f6e41df517922 P f49b0cd67c574a95aab9ad6aa430fda3: stopping tablet replica
I20260811 22:26:48.783205 17955 raft_consensus.cc:2229] T 3fc200fd6e184514b42f6e41df517922 P f49b0cd67c574a95aab9ad6aa430fda3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:26:48.783718 17955 raft_consensus.cc:2258] T 3fc200fd6e184514b42f6e41df517922 P f49b0cd67c574a95aab9ad6aa430fda3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:26:48.785320 17955 ts_tablet_manager.cc:1905] T 3fc200fd6e184514b42f6e41df517922 P f49b0cd67c574a95aab9ad6aa430fda3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:26:48.788591 17056 catalog_manager.cc:4841] TS 97edea8ccfb74184b1576d4868ff608a (127.29.106.3:36021): tablet 3fc200fd6e184514b42f6e41df517922 (table test-table [id=78080626114747e185e6d13af79adb49]) successfully deleted
I20260811 22:26:48.791417 17955 ts_tablet_manager.cc:1918] T 3fc200fd6e184514b42f6e41df517922 P f49b0cd67c574a95aab9ad6aa430fda3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:26:48.791564 17955 log.cc:1196] T 3fc200fd6e184514b42f6e41df517922 P f49b0cd67c574a95aab9ad6aa430fda3: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786487105757681-30120-0/minicluster-data/ts-1/wal/wals/3fc200fd6e184514b42f6e41df517922
I20260811 22:26:48.792053 17955 ts_tablet_manager.cc:1939] T 3fc200fd6e184514b42f6e41df517922 P f49b0cd67c574a95aab9ad6aa430fda3: Deleting consensus metadata
I20260811 22:26:48.793282 17056 catalog_manager.cc:4841] TS f49b0cd67c574a95aab9ad6aa430fda3 (127.29.106.2:34011): tablet 3fc200fd6e184514b42f6e41df517922 (table test-table [id=78080626114747e185e6d13af79adb49]) successfully deleted
I20260811 22:26:48.892004 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 17181
I20260811 22:26:48.920410 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 17397
Aug 11 22:26:48 dist-test-slave-gjrz krb5kdc[17001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.2@KRBTEST.COM
I20260811 22:26:48.934439 17906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:48.921180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56224 (local address 127.29.106.2:34011)
0811 22:26:48.921343 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:48.921347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:48.921417 (+ 70us) server_negotiation.cc:407] Connection header received
0811 22:26:48.921589 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:48.921593 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:48.921666 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:48.921740 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:48.923026 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:48.923847 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:48.925119 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:48.925505 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:48.925610 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:48.929495 (+ 3885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:48.929509 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:48.929526 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:48.929568 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:48.932139 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:48.932751 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:48.932758 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:48.932760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:48.932883 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:48.933221 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:48.933224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:48.933226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:48.933597 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:48.933812 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:48.934042 (+ 230us) server_negotiation.cc:299] Negotiation successful
0811 22:26:48.934197 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260811 22:26:48.935081 17627 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:26:49.021589 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 17397
I20260811 22:26:49.048444 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 17645
Aug 11 22:26:49 dist-test-slave-gjrz krb5kdc[17001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.3@KRBTEST.COM
I20260811 22:26:49.062703 17910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:26:49.049247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46462 (local address 127.29.106.3:36021)
0811 22:26:49.049439 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0811 22:26:49.049445 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:26:49.049466 (+ 21us) server_negotiation.cc:407] Connection header received
0811 22:26:49.049534 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:26:49.049540 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:26:49.049606 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:26:49.049691 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:26:49.050963 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:49.051739 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:49.052987 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:26:49.053390 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:26:49.053502 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:26:49.057533 (+ 4031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:26:49.057550 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:26:49.057568 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:26:49.057600 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:26:49.060007 (+ 2407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:49.060650 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:49.060657 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:49.060660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:49.060788 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:26:49.061219 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:26:49.061222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:26:49.061224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:26:49.061599 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:26:49.061897 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0811 22:26:49.062187 (+ 290us) server_negotiation.cc:299] Negotiation successful
0811 22:26:49.062393 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260811 22:26:49.063413 17876 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:26:49.150288 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 17645
I20260811 22:26:49.174297 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 17025
I20260811 22:26:49.175518 17130 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:26:49.284489 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 17025
2026-08-11T22:26:49Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4426 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:26:49 dist-test-slave-gjrz krb5kdc[17968](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:26:49 dist-test-slave-gjrz krb5kdc[17968](info): set up 2 sockets
Aug 11 22:26:49 dist-test-slave-gjrz krb5kdc[17968](info): commencing operation
krb5kdc: starting...
W20260811 22:26:50.555343 30120 mini_kdc.cc:121] Time spent starting KDC: real 1.222s user 0.000s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Aug 11 22:26:50 dist-test-slave-gjrz krb5kdc[17968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:26:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:26:50Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.29.106.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.29.106.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.29.106.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/krb5kdc/rangeradmin_127.29.106.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.29.106.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/krb5kdc/rangeradmin_127.29.106.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.29.106.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.29.106.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.29.106.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/krb5kdc/rangerlookup_127.29.106.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.29.106.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/krb5kdc/rangerlookup_127.29.106.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.29.106.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.29.106.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.29.106.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/krb5kdc/HTTP_127.29.106.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.29.106.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/krb5kdc/HTTP_127.29.106.20@KRBTEST.COM.keytab.
I20260811 22:26:50.891777 30120 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task7tIZk9/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/postgres -l logfile start
2026-08-11 22:26:53.193 UTC [18027] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-11 22:26:53.193 UTC [18027] LOG: listening on IPv4 address "127.29.106.20", port 50819
2026-08-11 22:26:53.196 UTC [18027] LOG: listening on Unix socket "/tmp/.s.PGSQL.50819"
2026-08-11 22:26:53.218 UTC [18039] LOG: database system was shut down at 2026-08-11 22:26:51 UTC
2026-08-11 22:26:53.222 UTC [18027] LOG: database system is ready to accept connections
I20260811 22:26:54.346212 30120 mini_postgres.cc:96] Postgres bound to 50819
2026-08-11 22:26:54.370 UTC [18054] FATAL: database "slave" does not exist
127.29.106.20:50819 - accepting connections
I20260811 22:26:54.371088 30120 mini_ranger.cc:161] Starting Ranger...
I20260811 22:26:54.403108 30120 mini_ranger.cc:84] Created miniranger Postgres user
I20260811 22:26:54.818717 30120 mini_ranger.cc:87] Created ranger Postgres database
I20260811 22:26:54.818873 30120 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ranger-admin
2026-08-11 22:26:55,185 [I] DB FLAVOR :POSTGRES
2026-08-11 22:26:55,185 [I] --------- Verifying Ranger DB connection ---------
2026-08-11 22:26:55,185 [I] Checking connection..
2026-08-11 22:26:55,185 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7tIZk9/build/asan/bin/postgresql.jar:/tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.106.20:50819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-11 22:26:55,446 [I] Checking connection passed.
2026-08-11 22:26:55,447 [I] --------- Verifying version history table ---------
2026-08-11 22:26:55,447 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7tIZk9/build/asan/bin/postgresql.jar:/tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.106.20:50819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-11 22:26:55,716 [I] Table x_db_version_h does not exist in database ranger
2026-08-11 22:26:55,716 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7tIZk9/build/asan/bin/postgresql.jar:/tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.106.20:50819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-11 22:26:55,966 [I] Table x_db_version_h does not exist in database ranger
2026-08-11 22:26:55,967 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-11 22:26:55,967 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7tIZk9/build/asan/bin/postgresql.jar:/tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.106.20:50819/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-11 22:26:56.209 UTC [18179] WARNING: there is no transaction in progress
2026-08-11 22:26:56,215 [I] create_dbversion_catalog.sql file imported successfully
2026-08-11 22:26:56,215 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7tIZk9/build/asan/bin/postgresql.jar:/tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.106.20:50819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-11 22:26:56,471 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-11 22:26:56,472 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-11 22:26:56,472 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7tIZk9/build/asan/bin/postgresql.jar:/tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.106.20:50819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-11 22:26:56,709 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7tIZk9/build/asan/bin/postgresql.jar:/tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.106.20:50819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-11 22:26:56,962 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7tIZk9/build/asan/bin/postgresql.jar:/tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.106.20:50819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-gjrz.c.gcp-upstream.internal','N') ;"
2026-08-11 22:26:57,182 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7tIZk9/build/asan/bin/postgresql.jar:/tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.106.20:50819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-11 22:26:57,439 [I] Table x_portal_user does not exist in database ranger
2026-08-11 22:26:57,440 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7tIZk9/build/asan/bin/postgresql.jar:/tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.106.20:50819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-11 22:26:57,672 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-11 22:26:57,672 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7tIZk9/build/asan/bin/postgresql.jar:/tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.106.20:50819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-11 22:26:57,906 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7tIZk9/build/asan/bin/postgresql.jar:/tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.106.20:50819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-11 22:26:58,160 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-11 22:26:58,161 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7tIZk9/build/asan/bin/postgresql.jar:/tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.106.20:50819/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-11 22:26:59.043 UTC [18373] WARNING: there is no transaction in progress
2026-08-11 22:26:59.059 UTC [18373] WARNING: there is no transaction in progress
2026-08-11 22:26:59.073 UTC [18373] WARNING: there is no transaction in progress
2026-08-11 22:26:59.087 UTC [18373] WARNING: there is no transaction in progress
2026-08-11 22:26:59.101 UTC [18373] WARNING: there is no transaction in progress
2026-08-11 22:26:59.120 UTC [18373] WARNING: there is no transaction in progress
2026-08-11 22:26:59.180 UTC [18373] WARNING: there is no transaction in progress
2026-08-11 22:26:59.186 UTC [18373] WARNING: there is no transaction in progress
2026-08-11 22:26:59.200 UTC [18373] WARNING: there is no transaction in progress
2026-08-11 22:26:59.210 UTC [18373] WARNING: there is no transaction in progress
2026-08-11 22:26:59.219 UTC [18373] WARNING: there is no transaction in progress
2026-08-11 22:26:59.231 UTC [18373] WARNING: there is no transaction in progress
2026-08-11 22:26:59.240 UTC [18373] WARNING: there is no transaction in progress
2026-08-11 22:26:59.246 UTC [18373] WARNING: there is no transaction in progress
2026-08-11 22:26:59.252 UTC [18373] WARNING: there is no transaction in progress
2026-08-11 22:26:59.257 UTC [18373] WARNING: there is no transaction in progress
2026-08-11 22:26:59,812 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-11 22:26:59,812 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7tIZk9/build/asan/bin/postgresql.jar:/tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.106.20:50819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-11 22:27:00,042 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-11 22:27:00,043 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7tIZk9/build/asan/bin/postgresql.jar:/tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.106.20:50819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-11 22:27:00,290 [I] Table x_portal_user already exists in database 'ranger'
2026-08-11 22:27:00,291 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7tIZk9/build/asan/bin/postgresql.jar:/tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.106.20:50819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-11 22:27:00,525 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-11 22:27:00,525 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7tIZk9/build/asan/bin/postgresql.jar:/tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.106.20:50819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-11 22:27:00,778 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7tIZk9/build/asan/bin/postgresql.jar:/tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.106.20:50819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-11 22:27:01,015 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7tIZk9/build/asan/bin/postgresql.jar:/tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.106.20:50819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-gjrz.c.gcp-upstream.internal';"
2026-08-11 22:27:01,242 [I] CORE_DB_SCHEMA import status has been updated
2026-08-11 22:27:01,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7tIZk9/build/asan/bin/postgresql.jar:/tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.106.20:50819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-11 22:27:01,492 [I] DB_PATCHES have already been applied
I20260811 22:27:01.499542 30120 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ranger-admin:/tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task7tIZk9/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task7tIZk9/build/asan/bin/postgresql.jar:/tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260811 22:27:01.499652 30120 mini_ranger.cc:193] Using host: 127.29.106.20
I20260811 22:27:01.520255 30120 mini_ranger.cc:227] Ranger admin URL: http://127.29.106.20:43757
Aug 11, 2026 10:27:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 11, 2026 10:27:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 11, 2026 10:27:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ranger-admin/ews/webapp
Aug 11, 2026 10:27:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 11, 2026 10:27:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 11, 2026 10:27:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ranger-admin/ews/webapp].
Aug 11, 2026 10:27:02 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260811 22:27:02.879520 30120 mini_ranger.cc:160] Time spent starting Ranger: real 8.508s user 0.000s sys 0.062s
Aug 11, 2026 10:27:22 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 11, 2026 10:27:22 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 11, 2026 10:27:22 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 11, 2026 10:27:22 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 11, 2026 10:27:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 11, 2026 10:27:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 11, 2026 10:27:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 11, 2026 10:27:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 11, 2026 10:27:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 11, 2026 10:27:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 11, 2026 10:27:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 11, 2026 10:27:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 11, 2026 10:27:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 11, 2026 10:27:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 11, 2026 10:27:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 11, 2026 10:27:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 11, 2026 10:27:22 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 11, 2026 10:27:23 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260811 22:27:24.385804 30120 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.29.106.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.29.106.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.29.106.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/krb5kdc/rangerkms_127.29.106.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.29.106.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/krb5kdc/rangerkms_127.29.106.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.29.106.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.29.106.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.29.106.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/krb5kdc/HTTP_127.29.106.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.29.106.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/krb5kdc/HTTP_127.29.106.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 11 22:27:24 dist-test-slave-gjrz krb5kdc[17968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487244, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260811 22:27:24.560546 30120 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260811 22:27:24.581727 30120 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260811 22:27:25.356196 30120 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260811 22:27:25.356338 30120 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ranger-kms
I20260811 22:27:25.356494 30120 mini_ranger_kms.cc:227] Using postgres at 127.29.106.20:50819
2026-08-11 22:27:25,885 [I] DB FLAVOR :POSTGRES
2026-08-11 22:27:25,886 [I] --------- Verifying Ranger DB connection ---------
2026-08-11 22:27:25,886 [I] Checking connection
2026-08-11 22:27:25,886 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7tIZk9/build/asan/bin/postgresql.jar:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.106.20:50819/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-11 22:27:26,115 [I] connection success
2026-08-11 22:27:26,115 [I] --------- Verifying Ranger DB tables ---------
2026-08-11 22:27:26,115 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-11 22:27:26,115 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7tIZk9/build/asan/bin/postgresql.jar:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.106.20:50819/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-11 22:27:26,340 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-11 22:27:26,341 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-11 22:27:26,341 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-11 22:27:26,341 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7tIZk9/build/asan/bin/postgresql.jar:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.106.20:50819/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-11 22:27:26,591 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260811 22:27:27.052796 30120 mini_ranger_kms.cc:310] Created kms service
I20260811 22:27:27.219779 30120 mini_ranger_kms.cc:326] Created kudu user
I20260811 22:27:27.260736 30120 mini_ranger_kms.cc:343] Created rangerkms user
Aug 11 22:27:27 dist-test-slave-gjrz krb5kdc[17968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487244, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.29.106.20@KRBTEST.COM
I20260811 22:27:27.638513 30120 mini_ranger_kms.cc:384] Added ranger policy
I20260811 22:27:27.641218 30120 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ranger-kms:/tmp/dist-test-task7tIZk9/build/asan/bin/postgresql.jar:/tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260811 22:27:27.641295 30120 mini_ranger_kms.cc:240] Using host: 127.29.106.20
I20260811 22:27:27.667398 30120 mini_ranger_kms.cc:276] Ranger KMS PID: 18735
I20260811 22:27:27.667675 30120 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.29.106.20:45157
Aug 11, 2026 10:27:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 11, 2026 10:27:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 11, 2026 10:27:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 11, 2026 10:27:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 11, 2026 10:27:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ranger-kms/ews/webapp].
Aug 11, 2026 10:27:29 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 11 22:27:29 dist-test-slave-gjrz krb5kdc[17968](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786487249, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.29.106.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 11 22:27:30 dist-test-slave-gjrz krb5kdc[17968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487249, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.29.106.20@KRBTEST.COM for HTTP/127.29.106.20@KRBTEST.COM
I20260811 22:27:30.351316 30120 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.791s user 0.145s sys 0.419s
I20260811 22:27:30.351639 30120 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260811 22:27:30.351851 30120 mini_ranger_kms.cc:401] 127.29.106.20:45157/kms/v1/keys
Aug 11 22:27:30 dist-test-slave-gjrz krb5kdc[17968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487249, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.29.106.20@KRBTEST.COM for HTTP/127.29.106.20@KRBTEST.COM
Aug 11 22:27:32 dist-test-slave-gjrz krb5kdc[17968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.29.106.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
I20260811 22:27:32.909278 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:39001
--webserver_interface=127.29.106.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:45101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:39001
--ranger_config_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.106.20:45157/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:27:33.517659 18818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:27:33.518056 18818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:27:33.518306 18818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:27:33.529613 18818 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:27:33.529848 18818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:27:33.530040 18818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:27:33.530246 18818 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:27:33.530406 18818 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:27:33.545439 18818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:45101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/master-0/wal
--ranger_kms_url=127.29.106.20:45157/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:39001
--ranger_config_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:39001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.106.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:27:33.546896 18818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:27:33.548509 18818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:27:33.567346 18827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:27:33.567808 18833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:27:33.567992 18828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:27:33.569574 18818 server_base.cc:1034] running on GCE node
I20260811 22:27:33.570451 18818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:27:33.572122 18818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:27:33.573379 18818 hybrid_clock.cc:648] HybridClock initialized: now 1786487253573282 us; error 29 us; skew 500 ppm
I20260811 22:27:33.573858 18818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:27:33 dist-test-slave-gjrz krb5kdc[17968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487253, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:27:33.584811 18818 init.cc:377] Logged in from keytab as kudu/127.29.106.62@KRBTEST.COM (short username kudu)
I20260811 22:27:33.587216 18818 webserver.cc:460] Webserver started at http://127.29.106.62:39781/ using document root <none> and password file <none>
I20260811 22:27:33.588181 18818 fs_manager.cc:359] Metadata directory not provided
I20260811 22:27:33.588402 18818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:27:33.588817 18818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 22:27:33 dist-test-slave-gjrz krb5kdc[17968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487253, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for HTTP/127.29.106.20@KRBTEST.COM
I20260811 22:27:33.664574 18818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "57f2ba315f6a4571a60fbab368f571a7"
format_stamp: "Formatted at 2026-08-11 22:27:33 on dist-test-slave-gjrz"
server_key: "8d8ffe7c8f27d7af4363b0b878333a7b"
server_key_iv: "959a51f576981c3843e4dee43d121e1f"
server_key_version: "kuduclusterkey@0"
I20260811 22:27:33.666083 18818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "57f2ba315f6a4571a60fbab368f571a7"
format_stamp: "Formatted at 2026-08-11 22:27:33 on dist-test-slave-gjrz"
server_key: "8d8ffe7c8f27d7af4363b0b878333a7b"
server_key_iv: "959a51f576981c3843e4dee43d121e1f"
server_key_version: "kuduclusterkey@0"
I20260811 22:27:33.676132 18818 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.006s sys 0.005s
I20260811 22:27:33.700563 18843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:27:33.702567 18818 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.003s
I20260811 22:27:33.702847 18818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "57f2ba315f6a4571a60fbab368f571a7"
format_stamp: "Formatted at 2026-08-11 22:27:33 on dist-test-slave-gjrz"
server_key: "8d8ffe7c8f27d7af4363b0b878333a7b"
server_key_iv: "959a51f576981c3843e4dee43d121e1f"
server_key_version: "kuduclusterkey@0"
I20260811 22:27:33.703233 18818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:27:33.725308 18818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:27:33.732141 18818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:27:33.732877 18818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:27:33.805334 18818 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.62:39001
I20260811 22:27:33.805505 18938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.62:39001 every 8 connection(s)
I20260811 22:27:33.807367 18818 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:27:33.813794 18939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:27:33.814111 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 18818
I20260811 22:27:33.814339 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:27:33.815168 30120 external_mini_cluster.cc:1428] Setting key a7a5d456a50dfd8569499a9252191051
I20260811 22:27:33.834621 18939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57f2ba315f6a4571a60fbab368f571a7: Bootstrap starting.
Aug 11 22:27:33 dist-test-slave-gjrz krb5kdc[17968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:27:33.840746 18939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57f2ba315f6a4571a60fbab368f571a7: Neither blocks nor log segments found. Creating new log.
I20260811 22:27:33.843242 18939 log.cc:824] T 00000000000000000000000000000000 P 57f2ba315f6a4571a60fbab368f571a7: Log is configured to *not* fsync() on all Append() calls
I20260811 22:27:33.847128 18945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:27:33.817628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39318 (local address 127.29.106.62:39001)
0811 22:27:33.819511 (+ 1883us) server_negotiation.cc:206] Beginning negotiation
0811 22:27:33.819529 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0811 22:27:33.819613 (+ 84us) server_negotiation.cc:407] Connection header received
0811 22:27:33.823932 (+ 4319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:27:33.823962 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:27:33.824545 (+ 583us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:27:33.825076 (+ 531us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:27:33.826302 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:27:33.829024 (+ 2722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:27:33.830394 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:27:33.831905 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:27:33.832108 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:27:33.836186 (+ 4078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:27:33.836201 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:27:33.836220 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:27:33.836271 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:27:33.842229 (+ 5958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:27:33.842958 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:27:33.842966 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:27:33.842974 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:27:33.843230 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:27:33.843646 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:27:33.843652 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:27:33.843655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:27:33.844592 (+ 937us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:27:33.844914 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0811 22:27:33.845408 (+ 494us) server_negotiation.cc:299] Negotiation successful
0811 22:27:33.845764 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1452,"thread_start_us":1061,"threads_started":1}
I20260811 22:27:33.847790 18939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57f2ba315f6a4571a60fbab368f571a7: No bootstrap required, opened a new log
I20260811 22:27:33.859907 18939 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57f2ba315f6a4571a60fbab368f571a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57f2ba315f6a4571a60fbab368f571a7" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 39001 } }
I20260811 22:27:33.860460 18939 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57f2ba315f6a4571a60fbab368f571a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:27:33.860620 18939 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57f2ba315f6a4571a60fbab368f571a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57f2ba315f6a4571a60fbab368f571a7, State: Initialized, Role: FOLLOWER
I20260811 22:27:33.861367 18939 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57f2ba315f6a4571a60fbab368f571a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57f2ba315f6a4571a60fbab368f571a7" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 39001 } }
I20260811 22:27:33.861682 18939 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57f2ba315f6a4571a60fbab368f571a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:27:33.861951 18939 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57f2ba315f6a4571a60fbab368f571a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:27:33.862211 18939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57f2ba315f6a4571a60fbab368f571a7 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:27:33.865654 18939 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57f2ba315f6a4571a60fbab368f571a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57f2ba315f6a4571a60fbab368f571a7" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 39001 } }
I20260811 22:27:33.866112 18939 leader_election.cc:302] T 00000000000000000000000000000000 P 57f2ba315f6a4571a60fbab368f571a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57f2ba315f6a4571a60fbab368f571a7; no voters:
I20260811 22:27:33.866982 18939 leader_election.cc:288] T 00000000000000000000000000000000 P 57f2ba315f6a4571a60fbab368f571a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:27:33.867350 18948 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57f2ba315f6a4571a60fbab368f571a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:27:33.869253 18948 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57f2ba315f6a4571a60fbab368f571a7 [term 1 LEADER]: Becoming Leader. State: Replica: 57f2ba315f6a4571a60fbab368f571a7, State: Running, Role: LEADER
I20260811 22:27:33.869942 18948 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57f2ba315f6a4571a60fbab368f571a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57f2ba315f6a4571a60fbab368f571a7" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 39001 } }
I20260811 22:27:33.870493 18939 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57f2ba315f6a4571a60fbab368f571a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:27:33.874286 18954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57f2ba315f6a4571a60fbab368f571a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57f2ba315f6a4571a60fbab368f571a7. Latest consensus state: current_term: 1 leader_uuid: "57f2ba315f6a4571a60fbab368f571a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57f2ba315f6a4571a60fbab368f571a7" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 39001 } } }
I20260811 22:27:33.874368 18951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57f2ba315f6a4571a60fbab368f571a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57f2ba315f6a4571a60fbab368f571a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57f2ba315f6a4571a60fbab368f571a7" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 39001 } } }
I20260811 22:27:33.875869 18954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57f2ba315f6a4571a60fbab368f571a7 [sys.catalog]: This master's current role is: LEADER
I20260811 22:27:33.875896 18951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57f2ba315f6a4571a60fbab368f571a7 [sys.catalog]: This master's current role is: LEADER
I20260811 22:27:33.886864 18963 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:27:33.897239 18963 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:27:33.898792 18939 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260811 22:27:33.920678 18963 catalog_manager.cc:1347] Generated new cluster ID: 8e90a26c8e6f4d92b4ecd2d9c0aeb6ec
I20260811 22:27:33.920811 18963 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:27:33.936757 18963 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:27:33.939332 18963 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:27:33.964881 18963 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57f2ba315f6a4571a60fbab368f571a7: Generated new TSK 0
I20260811 22:27:33.966024 18963 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.106.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab.
Entry for principal kudu/127.29.106.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab.
I20260811 22:27:34.838140 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.106.20:45157/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.1:0
--local_ip_for_outbound_sockets=127.29.106.1
--webserver_interface=127.29.106.1
--webserver_port=0
--tserver_master_addrs=127.29.106.62:39001
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab
--principal=kudu/127.29.106.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:45101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 22:27:35 dist-test-slave-gjrz krb5kdc[17968](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786487255, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260811 22:27:35.588053 19011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:27:35.588613 19011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:27:35.588979 19011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:27:35.605240 19011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:27:35.605466 19011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:27:35.605690 19011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.1
I20260811 22:27:35.626402 19011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:45101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.29.106.20:45157/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab
--principal=kudu/127.29.106.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.106.1
--webserver_port=0
--tserver_master_addrs=127.29.106.62:39001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.1
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:27:35.628509 19011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:27:35.630486 19011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Aug 11 22:27:35 dist-test-slave-gjrz krb5kdc[17968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487255, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for HTTP/127.29.106.20@KRBTEST.COM
W20260811 22:27:35.727526 19034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:27:35.733745 19029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:27:35.735522 19011 server_base.cc:1034] running on GCE node
W20260811 22:27:35.737046 19028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:27:35.740131 19011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:27:35.748194 19011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:27:35.752501 19011 hybrid_clock.cc:648] HybridClock initialized: now 1786487255752196 us; error 152 us; skew 500 ppm
I20260811 22:27:35.753106 19011 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:27:35 dist-test-slave-gjrz krb5kdc[17968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487255, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:27:35.758765 19011 init.cc:377] Logged in from keytab as kudu/127.29.106.1@KRBTEST.COM (short username kudu)
I20260811 22:27:35.770025 19011 webserver.cc:460] Webserver started at http://127.29.106.1:39419/ using document root <none> and password file <none>
I20260811 22:27:35.771041 19011 fs_manager.cc:359] Metadata directory not provided
I20260811 22:27:35.771198 19011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:27:35.771723 19011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 22:27:35 dist-test-slave-gjrz krb5kdc[17968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487255, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.1@KRBTEST.COM for HTTP/127.29.106.20@KRBTEST.COM
I20260811 22:27:35.796695 19011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-0/data/instance:
uuid: "01fa332a945a4f2589f1557320cdf416"
format_stamp: "Formatted at 2026-08-11 22:27:35 on dist-test-slave-gjrz"
server_key: "ee6e3376b0783cd3f7430a7492797a64"
server_key_iv: "2c5714622b9eed04bac81e15b5c81088"
server_key_version: "kuduclusterkey@0"
I20260811 22:27:35.798149 19011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance:
uuid: "01fa332a945a4f2589f1557320cdf416"
format_stamp: "Formatted at 2026-08-11 22:27:35 on dist-test-slave-gjrz"
server_key: "ee6e3376b0783cd3f7430a7492797a64"
server_key_iv: "2c5714622b9eed04bac81e15b5c81088"
server_key_version: "kuduclusterkey@0"
I20260811 22:27:35.827991 19011 fs_manager.cc:691] Time spent creating directory manager: real 0.029s user 0.005s sys 0.009s
I20260811 22:27:35.864588 19045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:27:35.866593 19011 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.005s sys 0.009s
I20260811 22:27:35.866868 19011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-0/wal
uuid: "01fa332a945a4f2589f1557320cdf416"
format_stamp: "Formatted at 2026-08-11 22:27:35 on dist-test-slave-gjrz"
server_key: "ee6e3376b0783cd3f7430a7492797a64"
server_key_iv: "2c5714622b9eed04bac81e15b5c81088"
server_key_version: "kuduclusterkey@0"
I20260811 22:27:35.867286 19011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:27:35.940317 19011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:27:35.946396 19011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:27:35.947139 19011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:27:35.964267 19011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:27:35.975787 19011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:27:35.976086 19011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260811 22:27:35.976483 19011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:27:35.976727 19011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 22:27:36 dist-test-slave-gjrz krb5kdc[17968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487255, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for HTTP/127.29.106.20@KRBTEST.COM
I20260811 22:27:36.569954 19011 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.1:45705
I20260811 22:27:36.570334 19249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.1:45705 every 8 connection(s)
I20260811 22:27:36.572084 19011 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
I20260811 22:27:36.575246 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 19011
I20260811 22:27:36.575529 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance
I20260811 22:27:36.576601 30120 external_mini_cluster.cc:1428] Setting key c444195c9a5216f9dd69205eb853504e
Aug 11 22:27:36 dist-test-slave-gjrz krb5kdc[17968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487255, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.1@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:27:36.630877 19268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:27:36.578155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:35521 (local address 127.29.106.62:39001)
0811 22:27:36.590130 (+ 11975us) server_negotiation.cc:206] Beginning negotiation
0811 22:27:36.590137 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:27:36.590162 (+ 25us) server_negotiation.cc:407] Connection header received
0811 22:27:36.595483 (+ 5321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:27:36.595490 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:27:36.595674 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:27:36.595996 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:27:36.599012 (+ 3016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:27:36.600604 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:27:36.603888 (+ 3284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:27:36.604871 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:27:36.605014 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:27:36.611676 (+ 6662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:27:36.611685 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:27:36.611690 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:27:36.611733 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:27:36.617630 (+ 5897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:27:36.624389 (+ 6759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:27:36.624396 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:27:36.624399 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:27:36.624703 (+ 304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:27:36.627716 (+ 3013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:27:36.627723 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:27:36.627726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:27:36.628262 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:27:36.629731 (+ 1469us) server_negotiation.cc:1035] Waiting for connection context
0811 22:27:36.629954 (+ 223us) server_negotiation.cc:299] Negotiation successful
0811 22:27:36.630207 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11783,"thread_start_us":11574,"threads_started":1}
I20260811 22:27:36.631796 19264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:27:36.577225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:39001 (local address 127.29.106.1:35521)
0811 22:27:36.587298 (+ 10073us) negotiation.cc:107] Waiting for socket to connect
0811 22:27:36.587363 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0811 22:27:36.593483 (+ 6120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:27:36.596349 (+ 2866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:27:36.596361 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:27:36.597425 (+ 1064us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:27:36.598755 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:27:36.598785 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:27:36.600819 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:27:36.600825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:27:36.603691 (+ 2866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:27:36.603708 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:27:36.605282 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:27:36.605295 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:27:36.605763 (+ 468us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:27:36.606865 (+ 1102us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:27:36.606900 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:27:36.611350 (+ 4450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:27:36.623367 (+ 12017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:27:36.623377 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:27:36.623395 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:27:36.624230 (+ 835us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:27:36.627104 (+ 2874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:27:36.627111 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:27:36.627115 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:27:36.627550 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:27:36.628464 (+ 914us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:27:36.628473 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:27:36.628961 (+ 488us) client_negotiation.cc:769] Sending connection context
0811 22:27:36.629338 (+ 377us) client_negotiation.cc:240] Negotiation successful
0811 22:27:36.629958 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9634,"thread_start_us":10949,"threads_started":1}
WARNING: no policy specified for kudu/127.29.106.2@KRBTEST.COM; defaulting to no policy
I20260811 22:27:36.634009 19258 heartbeater.cc:347] Connected to a master server at 127.29.106.62:39001
I20260811 22:27:36.634404 19258 heartbeater.cc:464] Registering TS with master...
I20260811 22:27:36.635752 19258 heartbeater.cc:511] Master 127.29.106.62:39001 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.2@KRBTEST.COM" created.
I20260811 22:27:36.638600 18869 ts_manager.cc:194] Registered new tserver with Master: 01fa332a945a4f2589f1557320cdf416 (127.29.106.1:45705)
I20260811 22:27:36.640776 18869 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.1@KRBTEST.COM'} at 127.29.106.1:35521
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab.
Entry for principal kudu/127.29.106.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab.
W20260811 22:27:36.685782 30120 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.29.106.2: real 0.105s user 0.000s sys 0.052s
I20260811 22:27:36.686493 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.106.20:45157/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.2:0
--local_ip_for_outbound_sockets=127.29.106.2
--webserver_interface=127.29.106.2
--webserver_port=0
--tserver_master_addrs=127.29.106.62:39001
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab
--principal=kudu/127.29.106.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:45101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:27:37.427116 19271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:27:37.427495 19271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:27:37.427727 19271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:27:37.438982 19271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:27:37.439183 19271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:27:37.439425 19271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.2
I20260811 22:27:37.453711 19271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:45101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.29.106.20:45157/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab
--principal=kudu/127.29.106.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.106.2
--webserver_port=0
--tserver_master_addrs=127.29.106.62:39001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.2
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:27:37.455128 19271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:27:37.456705 19271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:27:37.476428 19281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:27:37.476881 19285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:27:37.477233 19283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:27:37.478617 19271 server_base.cc:1034] running on GCE node
I20260811 22:27:37.479346 19271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:27:37.480490 19271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:27:37.481763 19271 hybrid_clock.cc:648] HybridClock initialized: now 1786487257481646 us; error 37 us; skew 500 ppm
I20260811 22:27:37.482221 19271 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:27:37 dist-test-slave-gjrz krb5kdc[17968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487257, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:27:37.486997 19271 init.cc:377] Logged in from keytab as kudu/127.29.106.2@KRBTEST.COM (short username kudu)
I20260811 22:27:37.488930 19271 webserver.cc:460] Webserver started at http://127.29.106.2:46857/ using document root <none> and password file <none>
I20260811 22:27:37.489753 19271 fs_manager.cc:359] Metadata directory not provided
I20260811 22:27:37.489945 19271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:27:37.490379 19271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 22:27:37 dist-test-slave-gjrz krb5kdc[17968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487257, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.2@KRBTEST.COM for HTTP/127.29.106.20@KRBTEST.COM
I20260811 22:27:37.511554 19271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-1/data/instance:
uuid: "c272d94669ef406185fc0d553c37685d"
format_stamp: "Formatted at 2026-08-11 22:27:37 on dist-test-slave-gjrz"
server_key: "a12049e4dd2e991549b363f6d8daedd8"
server_key_iv: "b2e98f6399a541db93533f226cedadb3"
server_key_version: "kuduclusterkey@0"
I20260811 22:27:37.512636 19271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance:
uuid: "c272d94669ef406185fc0d553c37685d"
format_stamp: "Formatted at 2026-08-11 22:27:37 on dist-test-slave-gjrz"
server_key: "a12049e4dd2e991549b363f6d8daedd8"
server_key_iv: "b2e98f6399a541db93533f226cedadb3"
server_key_version: "kuduclusterkey@0"
I20260811 22:27:37.520463 19271 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.002s
I20260811 22:27:37.540553 19294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:27:37.542394 19271 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260811 22:27:37.542652 19271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-1/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-1/wal
uuid: "c272d94669ef406185fc0d553c37685d"
format_stamp: "Formatted at 2026-08-11 22:27:37 on dist-test-slave-gjrz"
server_key: "a12049e4dd2e991549b363f6d8daedd8"
server_key_iv: "b2e98f6399a541db93533f226cedadb3"
server_key_version: "kuduclusterkey@0"
I20260811 22:27:37.543049 19271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:27:37.559368 19271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:27:37.564486 19271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:27:37.564983 19271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:27:37.566511 19271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:27:37.569300 19271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:27:37.569455 19271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:27:37.569707 19271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:27:37.569841 19271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:27:37.645035 19258 heartbeater.cc:503] Master 127.29.106.62:39001 was elected leader, sending a full tablet report...
I20260811 22:27:37.674304 19271 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.2:33711
I20260811 22:27:37.674433 19521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.2:33711 every 8 connection(s)
I20260811 22:27:37.675817 19271 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
I20260811 22:27:37.678347 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 19271
I20260811 22:27:37.678545 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance
I20260811 22:27:37.679103 30120 external_mini_cluster.cc:1428] Setting key 8b0a63cef704b33f639949dcf2f0c7f2
Aug 11 22:27:37 dist-test-slave-gjrz krb5kdc[17968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487257, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.2@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:27:37.709343 19527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:27:37.683018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:39309 (local address 127.29.106.62:39001)
0811 22:27:37.683831 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0811 22:27:37.683837 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:27:37.683859 (+ 22us) server_negotiation.cc:407] Connection header received
0811 22:27:37.687134 (+ 3275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:27:37.687139 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:27:37.687217 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:27:37.687598 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:27:37.689973 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:27:37.691663 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:27:37.694088 (+ 2425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:27:37.694962 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:27:37.695083 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:27:37.699997 (+ 4914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:27:37.700003 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:27:37.700008 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:27:37.700088 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:27:37.705515 (+ 5427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:27:37.706299 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:27:37.706304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:27:37.706308 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:27:37.706583 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:27:37.707089 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:27:37.707094 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:27:37.707097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:27:37.707634 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:27:37.707818 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0811 22:27:37.708762 (+ 944us) server_negotiation.cc:299] Negotiation successful
0811 22:27:37.708915 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":462,"threads_started":1}
I20260811 22:27:37.710747 19526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:27:37.681514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:39001 (local address 127.29.106.2:39309)
0811 22:27:37.683514 (+ 2000us) negotiation.cc:107] Waiting for socket to connect
0811 22:27:37.683609 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0811 22:27:37.686821 (+ 3212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:27:37.687838 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:27:37.687846 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:27:37.688604 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:27:37.689763 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:27:37.689789 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:27:37.691838 (+ 2049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:27:37.691842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:27:37.693944 (+ 2102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:27:37.693957 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:27:37.695348 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:27:37.695357 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:27:37.695697 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:27:37.696500 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:27:37.696566 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:27:37.699771 (+ 3205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:27:37.705642 (+ 5871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:27:37.705648 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:27:37.705659 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:27:37.706190 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:27:37.706686 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:27:37.706689 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:27:37.706691 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:27:37.706981 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:27:37.707807 (+ 826us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:27:37.707816 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:27:37.708302 (+ 486us) client_negotiation.cc:769] Sending connection context
0811 22:27:37.708676 (+ 374us) client_negotiation.cc:240] Negotiation successful
0811 22:27:37.709220 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1416,"thread_start_us":1975,"threads_started":1}
I20260811 22:27:37.712853 19523 heartbeater.cc:347] Connected to a master server at 127.29.106.62:39001
I20260811 22:27:37.713209 19523 heartbeater.cc:464] Registering TS with master...
I20260811 22:27:37.714377 19523 heartbeater.cc:511] Master 127.29.106.62:39001 requested a full tablet report, sending...
I20260811 22:27:37.716374 18869 ts_manager.cc:194] Registered new tserver with Master: c272d94669ef406185fc0d553c37685d (127.29.106.2:33711)
I20260811 22:27:37.717280 18869 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.2@KRBTEST.COM'} at 127.29.106.2:39309
WARNING: no policy specified for kudu/127.29.106.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab.
Entry for principal kudu/127.29.106.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab.
I20260811 22:27:37.766381 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.106.20:45157/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.3:0
--local_ip_for_outbound_sockets=127.29.106.3
--webserver_interface=127.29.106.3
--webserver_port=0
--tserver_master_addrs=127.29.106.62:39001
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab
--principal=kudu/127.29.106.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:45101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:27:38.463363 19531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:27:38.463716 19531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:27:38.463934 19531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:27:38.475643 19531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:27:38.475871 19531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:27:38.476141 19531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.3
I20260811 22:27:38.491422 19531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:45101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.29.106.20:45157/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab
--principal=kudu/127.29.106.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.106.3
--webserver_port=0
--tserver_master_addrs=127.29.106.62:39001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.3
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:27:38.493382 19531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:27:38.495504 19531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:27:38.517125 19540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:27:38.518170 19541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:27:38.518184 19552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:27:38.519472 19531 server_base.cc:1034] running on GCE node
I20260811 22:27:38.520320 19531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:27:38.521696 19531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:27:38.523030 19531 hybrid_clock.cc:648] HybridClock initialized: now 1786487258522902 us; error 38 us; skew 500 ppm
I20260811 22:27:38.523497 19531 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:27:38 dist-test-slave-gjrz krb5kdc[17968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487258, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:27:38.528131 19531 init.cc:377] Logged in from keytab as kudu/127.29.106.3@KRBTEST.COM (short username kudu)
I20260811 22:27:38.530416 19531 webserver.cc:460] Webserver started at http://127.29.106.3:43401/ using document root <none> and password file <none>
I20260811 22:27:38.531430 19531 fs_manager.cc:359] Metadata directory not provided
I20260811 22:27:38.531690 19531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:27:38.532163 19531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 22:27:38 dist-test-slave-gjrz krb5kdc[17968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487258, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.3@KRBTEST.COM for HTTP/127.29.106.20@KRBTEST.COM
I20260811 22:27:38.548002 19531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-2/data/instance:
uuid: "d4653108eeb0438abdabf77a0f37ae26"
format_stamp: "Formatted at 2026-08-11 22:27:38 on dist-test-slave-gjrz"
server_key: "708e92caef1a53a6106f1dcaeab948cc"
server_key_iv: "063f952843f0361a6002b33b6f2966ab"
server_key_version: "kuduclusterkey@0"
I20260811 22:27:38.549153 19531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance:
uuid: "d4653108eeb0438abdabf77a0f37ae26"
format_stamp: "Formatted at 2026-08-11 22:27:38 on dist-test-slave-gjrz"
server_key: "708e92caef1a53a6106f1dcaeab948cc"
server_key_iv: "063f952843f0361a6002b33b6f2966ab"
server_key_version: "kuduclusterkey@0"
I20260811 22:27:38.556988 19531 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.006s
I20260811 22:27:38.575685 19560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:27:38.577445 19531 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.003s
I20260811 22:27:38.577644 19531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-2/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-2/wal
uuid: "d4653108eeb0438abdabf77a0f37ae26"
format_stamp: "Formatted at 2026-08-11 22:27:38 on dist-test-slave-gjrz"
server_key: "708e92caef1a53a6106f1dcaeab948cc"
server_key_iv: "063f952843f0361a6002b33b6f2966ab"
server_key_version: "kuduclusterkey@0"
I20260811 22:27:38.577914 19531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:27:38.609055 19531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:27:38.615104 19531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:27:38.615756 19531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:27:38.617702 19531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:27:38.620503 19531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:27:38.620671 19531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:27:38.620910 19531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:27:38.621057 19531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:27:38.720366 19523 heartbeater.cc:503] Master 127.29.106.62:39001 was elected leader, sending a full tablet report...
I20260811 22:27:38.733227 19531 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.3:32869
I20260811 22:27:38.733384 19775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.3:32869 every 8 connection(s)
I20260811 22:27:38.734905 19531 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
I20260811 22:27:38.736886 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 19531
I20260811 22:27:38.737063 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance
I20260811 22:27:38.737588 30120 external_mini_cluster.cc:1428] Setting key 5aa4b8e0c530798c3a4537e0c09362e6
Aug 11 22:27:38 dist-test-slave-gjrz krb5kdc[17968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487258, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.3@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:27:38.764348 19782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:27:38.739161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:54127 (local address 127.29.106.62:39001)
0811 22:27:38.740131 (+ 970us) server_negotiation.cc:206] Beginning negotiation
0811 22:27:38.740136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:27:38.742418 (+ 2282us) server_negotiation.cc:407] Connection header received
0811 22:27:38.745691 (+ 3273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:27:38.745697 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:27:38.745810 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:27:38.745911 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:27:38.748083 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:27:38.749206 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:27:38.751560 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:27:38.752419 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:27:38.752598 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:27:38.757171 (+ 4573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:27:38.757176 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:27:38.757180 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:27:38.757242 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:27:38.761095 (+ 3853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:27:38.761892 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:27:38.761897 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:27:38.761899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:27:38.762071 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:27:38.762534 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:27:38.762537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:27:38.762539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:27:38.762894 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:27:38.763040 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0811 22:27:38.763686 (+ 646us) server_negotiation.cc:299] Negotiation successful
0811 22:27:38.763838 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":837,"thread_start_us":645,"threads_started":1}
I20260811 22:27:38.765439 19783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:27:38.740326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:39001 (local address 127.29.106.3:54127)
0811 22:27:38.742250 (+ 1924us) negotiation.cc:107] Waiting for socket to connect
0811 22:27:38.742300 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0811 22:27:38.745444 (+ 3144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:27:38.746190 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:27:38.746198 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:27:38.746918 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:27:38.747916 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:27:38.747936 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:27:38.749343 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:27:38.749347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:27:38.751383 (+ 2036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:27:38.751396 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:27:38.752776 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:27:38.752787 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:27:38.753198 (+ 411us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:27:38.754069 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:27:38.754106 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:27:38.756920 (+ 2814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:27:38.761219 (+ 4299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:27:38.761226 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:27:38.761238 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:27:38.761745 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:27:38.762161 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:27:38.762164 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:27:38.762167 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:27:38.762447 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:27:38.762989 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:27:38.762994 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:27:38.763339 (+ 345us) client_negotiation.cc:769] Sending connection context
0811 22:27:38.763586 (+ 247us) client_negotiation.cc:240] Negotiation successful
0811 22:27:38.763967 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1604,"thread_start_us":1044,"threads_started":1}
I20260811 22:27:38.767131 19776 heartbeater.cc:347] Connected to a master server at 127.29.106.62:39001
I20260811 22:27:38.767514 19776 heartbeater.cc:464] Registering TS with master...
I20260811 22:27:38.768491 19776 heartbeater.cc:511] Master 127.29.106.62:39001 requested a full tablet report, sending...
I20260811 22:27:38.770403 18869 ts_manager.cc:194] Registered new tserver with Master: d4653108eeb0438abdabf77a0f37ae26 (127.29.106.3:32869)
I20260811 22:27:38.771495 18869 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.3@KRBTEST.COM'} at 127.29.106.3:54127
I20260811 22:27:38.778159 30120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 22:27:38.802170 19782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:27:38.784388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39330 (local address 127.29.106.62:39001)
0811 22:27:38.784527 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0811 22:27:38.784531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:27:38.785440 (+ 909us) server_negotiation.cc:407] Connection header received
0811 22:27:38.785618 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:27:38.785624 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:27:38.785736 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:27:38.785817 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:27:38.787843 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:27:38.788878 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:27:38.791015 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:27:38.791827 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:27:38.791938 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:27:38.795584 (+ 3646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:27:38.795590 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:27:38.795594 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:27:38.795624 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:27:38.798871 (+ 3247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:27:38.799618 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:27:38.799624 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:27:38.799628 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:27:38.799825 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:27:38.800310 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:27:38.800315 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:27:38.800318 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:27:38.800676 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:27:38.800995 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0811 22:27:38.801408 (+ 413us) server_negotiation.cc:299] Negotiation successful
0811 22:27:38.801668 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260811 22:27:38.806617 18869 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:27:38.820946 18869 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:27:38.865428 19805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:27:38.839633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.2:33711 (local address 127.0.0.1:45684)
0811 22:27:38.841904 (+ 2271us) negotiation.cc:107] Waiting for socket to connect
0811 22:27:38.841944 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0811 22:27:38.842202 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:27:38.852044 (+ 9842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:27:38.852049 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:27:38.852071 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:27:38.853314 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:27:38.853330 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:27:38.856694 (+ 3364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:27:38.856703 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:27:38.860877 (+ 4174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:27:38.860891 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:27:38.863903 (+ 3012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:27:38.863912 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:27:38.864219 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:27:38.864277 (+ 58us) client_negotiation.cc:769] Sending connection context
0811 22:27:38.864441 (+ 164us) client_negotiation.cc:240] Negotiation successful
0811 22:27:38.864636 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1934,"thread_start_us":18986,"threads_started":1}
I20260811 22:27:38.865478 19800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:27:38.839689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45684 (local address 127.29.106.2:33711)
0811 22:27:38.851373 (+ 11684us) server_negotiation.cc:206] Beginning negotiation
0811 22:27:38.851384 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 22:27:38.851415 (+ 31us) server_negotiation.cc:407] Connection header received
0811 22:27:38.851525 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:27:38.851534 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:27:38.851795 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:27:38.853563 (+ 1768us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:27:38.854005 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:27:38.856478 (+ 2473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:27:38.861112 (+ 4634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:27:38.863480 (+ 2368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:27:38.863811 (+ 331us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:27:38.864215 (+ 404us) server_negotiation.cc:1035] Waiting for connection context
0811 22:27:38.864451 (+ 236us) server_negotiation.cc:299] Negotiation successful
0811 22:27:38.864725 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11384,"thread_start_us":11142,"threads_started":1}
I20260811 22:27:38.868948 19810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:27:38.840267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:45705 (local address 127.0.0.1:42716)
0811 22:27:38.843589 (+ 3322us) negotiation.cc:107] Waiting for socket to connect
0811 22:27:38.843608 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 22:27:38.843762 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:27:38.849931 (+ 6169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:27:38.849939 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:27:38.849974 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:27:38.851638 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:27:38.851653 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:27:38.855674 (+ 4021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:27:38.855679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:27:38.862026 (+ 6347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:27:38.862045 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:27:38.867624 (+ 5579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:27:38.867635 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:27:38.867925 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:27:38.867972 (+ 47us) client_negotiation.cc:769] Sending connection context
0811 22:27:38.868074 (+ 102us) client_negotiation.cc:240] Negotiation successful
0811 22:27:38.868213 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3145,"mutex_wait_us":47,"thread_start_us":2244,"threads_started":1}
I20260811 22:27:38.869740 19807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:27:38.841862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42716 (local address 127.29.106.1:45705)
0811 22:27:38.847965 (+ 6103us) server_negotiation.cc:206] Beginning negotiation
0811 22:27:38.847975 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 22:27:38.848008 (+ 33us) server_negotiation.cc:407] Connection header received
0811 22:27:38.848159 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:27:38.848168 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:27:38.848480 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:27:38.852001 (+ 3521us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:27:38.852521 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:27:38.855528 (+ 3007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:27:38.863424 (+ 7896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:27:38.866491 (+ 3067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:27:38.866705 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:27:38.867071 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0811 22:27:38.868379 (+ 1308us) server_negotiation.cc:299] Negotiation successful
0811 22:27:38.869023 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5870,"thread_start_us":5638,"threads_started":1}
I20260811 22:27:38.870846 19389 tablet_service.cc:1467] Processing CreateTablet for tablet 604dbfa49eb747fb80692db93d6e5719 (DEFAULT_TABLE table=test-table [id=9e81ec0393824482aec52300e67f80ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:27:38.874909 19389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 604dbfa49eb747fb80692db93d6e5719. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:27:38.874668 19136 tablet_service.cc:1467] Processing CreateTablet for tablet 604dbfa49eb747fb80692db93d6e5719 (DEFAULT_TABLE table=test-table [id=9e81ec0393824482aec52300e67f80ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:27:38.876960 19136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 604dbfa49eb747fb80692db93d6e5719. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:27:38.879007 19806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:27:38.841207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39378 (local address 127.29.106.3:32869)
0811 22:27:38.850200 (+ 8993us) server_negotiation.cc:206] Beginning negotiation
0811 22:27:38.850208 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 22:27:38.856388 (+ 6180us) server_negotiation.cc:407] Connection header received
0811 22:27:38.856823 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:27:38.856833 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:27:38.857125 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:27:38.857373 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:27:38.861158 (+ 3785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:27:38.863501 (+ 2343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:27:38.872790 (+ 9289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:27:38.875589 (+ 2799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:27:38.875727 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:27:38.876057 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0811 22:27:38.878055 (+ 1998us) server_negotiation.cc:299] Negotiation successful
0811 22:27:38.878278 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8756,"thread_start_us":8340,"threads_started":1}
I20260811 22:27:38.879834 19803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:27:38.842073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.3:32869 (local address 127.0.0.1:39378)
0811 22:27:38.854578 (+ 12505us) negotiation.cc:107] Waiting for socket to connect
0811 22:27:38.854597 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 22:27:38.856545 (+ 1948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:27:38.858147 (+ 1602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:27:38.858153 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:27:38.858179 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:27:38.860734 (+ 2555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:27:38.860752 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:27:38.865736 (+ 4984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:27:38.865742 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:27:38.872635 (+ 6893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:27:38.872650 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:27:38.876412 (+ 3762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:27:38.876422 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:27:38.877782 (+ 1360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:27:38.877819 (+ 37us) client_negotiation.cc:769] Sending connection context
0811 22:27:38.878837 (+ 1018us) client_negotiation.cc:240] Negotiation successful
0811 22:27:38.878991 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12322,"thread_start_us":1241,"threads_started":1}
I20260811 22:27:38.884203 19643 tablet_service.cc:1467] Processing CreateTablet for tablet 604dbfa49eb747fb80692db93d6e5719 (DEFAULT_TABLE table=test-table [id=9e81ec0393824482aec52300e67f80ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:27:38.885946 19643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 604dbfa49eb747fb80692db93d6e5719. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:27:38.898253 19814 tablet_bootstrap.cc:492] T 604dbfa49eb747fb80692db93d6e5719 P c272d94669ef406185fc0d553c37685d: Bootstrap starting.
I20260811 22:27:38.901036 19815 tablet_bootstrap.cc:492] T 604dbfa49eb747fb80692db93d6e5719 P 01fa332a945a4f2589f1557320cdf416: Bootstrap starting.
I20260811 22:27:38.905306 19818 tablet_bootstrap.cc:492] T 604dbfa49eb747fb80692db93d6e5719 P d4653108eeb0438abdabf77a0f37ae26: Bootstrap starting.
I20260811 22:27:38.907047 19814 tablet_bootstrap.cc:654] T 604dbfa49eb747fb80692db93d6e5719 P c272d94669ef406185fc0d553c37685d: Neither blocks nor log segments found. Creating new log.
I20260811 22:27:38.907459 19815 tablet_bootstrap.cc:654] T 604dbfa49eb747fb80692db93d6e5719 P 01fa332a945a4f2589f1557320cdf416: Neither blocks nor log segments found. Creating new log.
I20260811 22:27:38.909210 19814 log.cc:824] T 604dbfa49eb747fb80692db93d6e5719 P c272d94669ef406185fc0d553c37685d: Log is configured to *not* fsync() on all Append() calls
I20260811 22:27:38.910053 19815 log.cc:824] T 604dbfa49eb747fb80692db93d6e5719 P 01fa332a945a4f2589f1557320cdf416: Log is configured to *not* fsync() on all Append() calls
I20260811 22:27:38.912066 19818 tablet_bootstrap.cc:654] T 604dbfa49eb747fb80692db93d6e5719 P d4653108eeb0438abdabf77a0f37ae26: Neither blocks nor log segments found. Creating new log.
I20260811 22:27:38.914760 19818 log.cc:824] T 604dbfa49eb747fb80692db93d6e5719 P d4653108eeb0438abdabf77a0f37ae26: Log is configured to *not* fsync() on all Append() calls
I20260811 22:27:38.914829 19815 tablet_bootstrap.cc:492] T 604dbfa49eb747fb80692db93d6e5719 P 01fa332a945a4f2589f1557320cdf416: No bootstrap required, opened a new log
I20260811 22:27:38.914865 19814 tablet_bootstrap.cc:492] T 604dbfa49eb747fb80692db93d6e5719 P c272d94669ef406185fc0d553c37685d: No bootstrap required, opened a new log
I20260811 22:27:38.915234 19815 ts_tablet_manager.cc:1397] T 604dbfa49eb747fb80692db93d6e5719 P 01fa332a945a4f2589f1557320cdf416: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.005s
I20260811 22:27:38.915302 19814 ts_tablet_manager.cc:1397] T 604dbfa49eb747fb80692db93d6e5719 P c272d94669ef406185fc0d553c37685d: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.008s
I20260811 22:27:38.920054 19818 tablet_bootstrap.cc:492] T 604dbfa49eb747fb80692db93d6e5719 P d4653108eeb0438abdabf77a0f37ae26: No bootstrap required, opened a new log
I20260811 22:27:38.920495 19818 ts_tablet_manager.cc:1397] T 604dbfa49eb747fb80692db93d6e5719 P d4653108eeb0438abdabf77a0f37ae26: Time spent bootstrapping tablet: real 0.015s user 0.013s sys 0.000s
I20260811 22:27:38.928755 19815 raft_consensus.cc:348] T 604dbfa49eb747fb80692db93d6e5719 P 01fa332a945a4f2589f1557320cdf416 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c272d94669ef406185fc0d553c37685d" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 33711 } } peers { permanent_uuid: "d4653108eeb0438abdabf77a0f37ae26" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 32869 } } peers { permanent_uuid: "01fa332a945a4f2589f1557320cdf416" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 45705 } }
I20260811 22:27:38.929294 19815 raft_consensus.cc:374] T 604dbfa49eb747fb80692db93d6e5719 P 01fa332a945a4f2589f1557320cdf416 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:27:38.929523 19815 raft_consensus.cc:729] T 604dbfa49eb747fb80692db93d6e5719 P 01fa332a945a4f2589f1557320cdf416 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01fa332a945a4f2589f1557320cdf416, State: Initialized, Role: FOLLOWER
I20260811 22:27:38.930226 19815 consensus_queue.cc:260] T 604dbfa49eb747fb80692db93d6e5719 P 01fa332a945a4f2589f1557320cdf416 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c272d94669ef406185fc0d553c37685d" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 33711 } } peers { permanent_uuid: "d4653108eeb0438abdabf77a0f37ae26" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 32869 } } peers { permanent_uuid: "01fa332a945a4f2589f1557320cdf416" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 45705 } }
I20260811 22:27:38.932011 19814 raft_consensus.cc:348] T 604dbfa49eb747fb80692db93d6e5719 P c272d94669ef406185fc0d553c37685d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c272d94669ef406185fc0d553c37685d" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 33711 } } peers { permanent_uuid: "d4653108eeb0438abdabf77a0f37ae26" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 32869 } } peers { permanent_uuid: "01fa332a945a4f2589f1557320cdf416" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 45705 } }
I20260811 22:27:38.932621 19814 raft_consensus.cc:374] T 604dbfa49eb747fb80692db93d6e5719 P c272d94669ef406185fc0d553c37685d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:27:38.932793 19814 raft_consensus.cc:729] T 604dbfa49eb747fb80692db93d6e5719 P c272d94669ef406185fc0d553c37685d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c272d94669ef406185fc0d553c37685d, State: Initialized, Role: FOLLOWER
I20260811 22:27:38.933089 19815 ts_tablet_manager.cc:1428] T 604dbfa49eb747fb80692db93d6e5719 P 01fa332a945a4f2589f1557320cdf416: Time spent starting tablet: real 0.018s user 0.002s sys 0.014s
I20260811 22:27:38.933645 19814 consensus_queue.cc:260] T 604dbfa49eb747fb80692db93d6e5719 P c272d94669ef406185fc0d553c37685d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c272d94669ef406185fc0d553c37685d" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 33711 } } peers { permanent_uuid: "d4653108eeb0438abdabf77a0f37ae26" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 32869 } } peers { permanent_uuid: "01fa332a945a4f2589f1557320cdf416" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 45705 } }
I20260811 22:27:38.936287 19814 ts_tablet_manager.cc:1428] T 604dbfa49eb747fb80692db93d6e5719 P c272d94669ef406185fc0d553c37685d: Time spent starting tablet: real 0.021s user 0.009s sys 0.011s
I20260811 22:27:38.938409 19818 raft_consensus.cc:348] T 604dbfa49eb747fb80692db93d6e5719 P d4653108eeb0438abdabf77a0f37ae26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c272d94669ef406185fc0d553c37685d" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 33711 } } peers { permanent_uuid: "d4653108eeb0438abdabf77a0f37ae26" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 32869 } } peers { permanent_uuid: "01fa332a945a4f2589f1557320cdf416" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 45705 } }
I20260811 22:27:38.939042 19818 raft_consensus.cc:374] T 604dbfa49eb747fb80692db93d6e5719 P d4653108eeb0438abdabf77a0f37ae26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:27:38.939193 19818 raft_consensus.cc:729] T 604dbfa49eb747fb80692db93d6e5719 P d4653108eeb0438abdabf77a0f37ae26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4653108eeb0438abdabf77a0f37ae26, State: Initialized, Role: FOLLOWER
I20260811 22:27:38.939898 19818 consensus_queue.cc:260] T 604dbfa49eb747fb80692db93d6e5719 P d4653108eeb0438abdabf77a0f37ae26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c272d94669ef406185fc0d553c37685d" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 33711 } } peers { permanent_uuid: "d4653108eeb0438abdabf77a0f37ae26" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 32869 } } peers { permanent_uuid: "01fa332a945a4f2589f1557320cdf416" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 45705 } }
I20260811 22:27:38.941798 19776 heartbeater.cc:503] Master 127.29.106.62:39001 was elected leader, sending a full tablet report...
I20260811 22:27:38.942420 19818 ts_tablet_manager.cc:1428] T 604dbfa49eb747fb80692db93d6e5719 P d4653108eeb0438abdabf77a0f37ae26: Time spent starting tablet: real 0.022s user 0.009s sys 0.012s
W20260811 22:27:38.988090 19778 tablet.cc:2369] T 604dbfa49eb747fb80692db93d6e5719 P d4653108eeb0438abdabf77a0f37ae26: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:27:39.007053 19829 raft_consensus.cc:482] T 604dbfa49eb747fb80692db93d6e5719 P c272d94669ef406185fc0d553c37685d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:27:39.007627 19829 raft_consensus.cc:504] T 604dbfa49eb747fb80692db93d6e5719 P c272d94669ef406185fc0d553c37685d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c272d94669ef406185fc0d553c37685d" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 33711 } } peers { permanent_uuid: "d4653108eeb0438abdabf77a0f37ae26" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 32869 } } peers { permanent_uuid: "01fa332a945a4f2589f1557320cdf416" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 45705 } }
I20260811 22:27:39.010521 19829 leader_election.cc:288] T 604dbfa49eb747fb80692db93d6e5719 P c272d94669ef406185fc0d553c37685d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4653108eeb0438abdabf77a0f37ae26 (127.29.106.3:32869), 01fa332a945a4f2589f1557320cdf416 (127.29.106.1:45705)
I20260811 22:27:39.023910 19806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:27:39.011132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:43405 (local address 127.29.106.3:32869)
0811 22:27:39.011415 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0811 22:27:39.011420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:27:39.013430 (+ 2010us) server_negotiation.cc:407] Connection header received
0811 22:27:39.013794 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:27:39.013800 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:27:39.013878 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:27:39.013942 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:27:39.015639 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:27:39.017106 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:27:39.020585 (+ 3479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:27:39.022436 (+ 1851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:27:39.022599 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:27:39.022656 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0811 22:27:39.023315 (+ 659us) server_negotiation.cc:299] Negotiation successful
0811 22:27:39.023503 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260811 22:27:39.024155 19834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:27:39.010471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.3:32869 (local address 127.29.106.2:43405)
0811 22:27:39.013314 (+ 2843us) negotiation.cc:107] Waiting for socket to connect
0811 22:27:39.013338 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0811 22:27:39.013606 (+ 268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:27:39.014291 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:27:39.014294 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:27:39.014316 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:27:39.015455 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:27:39.015469 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:27:39.017275 (+ 1806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:27:39.017280 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:27:39.020418 (+ 3138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:27:39.020431 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:27:39.022816 (+ 2385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:27:39.022830 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:27:39.023070 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:27:39.023109 (+ 39us) client_negotiation.cc:769] Sending connection context
0811 22:27:39.023189 (+ 80us) client_negotiation.cc:240] Negotiation successful
0811 22:27:39.023354 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2582,"mutex_wait_us":3,"thread_start_us":1936,"threads_started":1}
I20260811 22:27:39.024580 19807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:27:39.011411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:38559 (local address 127.29.106.1:45705)
0811 22:27:39.011639 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0811 22:27:39.011643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:27:39.014432 (+ 2789us) server_negotiation.cc:407] Connection header received
0811 22:27:39.014629 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:27:39.014634 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:27:39.014713 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:27:39.014827 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:27:39.016578 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:27:39.017984 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:27:39.021581 (+ 3597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:27:39.023080 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:27:39.023301 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:27:39.023371 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0811 22:27:39.024155 (+ 784us) server_negotiation.cc:299] Negotiation successful
0811 22:27:39.024292 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260811 22:27:39.025442 19837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:27:39.012228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:45705 (local address 127.29.106.2:38559)
0811 22:27:39.014319 (+ 2091us) negotiation.cc:107] Waiting for socket to connect
0811 22:27:39.014338 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 22:27:39.014496 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:27:39.014836 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:27:39.014840 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:27:39.014860 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:27:39.016290 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:27:39.016306 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:27:39.018163 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:27:39.018167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:27:39.021383 (+ 3216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:27:39.021400 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:27:39.023630 (+ 2230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:27:39.023639 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:27:39.023940 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:27:39.023973 (+ 33us) client_negotiation.cc:769] Sending connection context
0811 22:27:39.024756 (+ 783us) client_negotiation.cc:240] Negotiation successful
0811 22:27:39.024896 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1793,"thread_start_us":1336,"threads_started":1}
I20260811 22:27:39.025709 19685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604dbfa49eb747fb80692db93d6e5719" candidate_uuid: "c272d94669ef406185fc0d553c37685d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4653108eeb0438abdabf77a0f37ae26" is_pre_election: true
I20260811 22:27:39.026777 19685 raft_consensus.cc:2454] T 604dbfa49eb747fb80692db93d6e5719 P d4653108eeb0438abdabf77a0f37ae26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c272d94669ef406185fc0d553c37685d in term 0.
I20260811 22:27:39.027205 19170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604dbfa49eb747fb80692db93d6e5719" candidate_uuid: "c272d94669ef406185fc0d553c37685d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01fa332a945a4f2589f1557320cdf416" is_pre_election: true
I20260811 22:27:39.027904 19310 leader_election.cc:302] T 604dbfa49eb747fb80692db93d6e5719 P c272d94669ef406185fc0d553c37685d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c272d94669ef406185fc0d553c37685d, d4653108eeb0438abdabf77a0f37ae26; no voters:
I20260811 22:27:39.028277 19170 raft_consensus.cc:2454] T 604dbfa49eb747fb80692db93d6e5719 P 01fa332a945a4f2589f1557320cdf416 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c272d94669ef406185fc0d553c37685d in term 0.
I20260811 22:27:39.028565 19829 raft_consensus.cc:2781] T 604dbfa49eb747fb80692db93d6e5719 P c272d94669ef406185fc0d553c37685d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:27:39.028888 19829 raft_consensus.cc:482] T 604dbfa49eb747fb80692db93d6e5719 P c272d94669ef406185fc0d553c37685d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:27:39.029098 19829 raft_consensus.cc:3037] T 604dbfa49eb747fb80692db93d6e5719 P c272d94669ef406185fc0d553c37685d [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:27:39.074592 19829 raft_consensus.cc:504] T 604dbfa49eb747fb80692db93d6e5719 P c272d94669ef406185fc0d553c37685d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c272d94669ef406185fc0d553c37685d" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 33711 } } peers { permanent_uuid: "d4653108eeb0438abdabf77a0f37ae26" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 32869 } } peers { permanent_uuid: "01fa332a945a4f2589f1557320cdf416" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 45705 } }
I20260811 22:27:39.075544 19829 leader_election.cc:288] T 604dbfa49eb747fb80692db93d6e5719 P c272d94669ef406185fc0d553c37685d [CANDIDATE]: Term 1 election: Requested vote from peers d4653108eeb0438abdabf77a0f37ae26 (127.29.106.3:32869), 01fa332a945a4f2589f1557320cdf416 (127.29.106.1:45705)
I20260811 22:27:39.076203 19685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604dbfa49eb747fb80692db93d6e5719" candidate_uuid: "c272d94669ef406185fc0d553c37685d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4653108eeb0438abdabf77a0f37ae26"
I20260811 22:27:39.076329 19170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604dbfa49eb747fb80692db93d6e5719" candidate_uuid: "c272d94669ef406185fc0d553c37685d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01fa332a945a4f2589f1557320cdf416"
I20260811 22:27:39.076452 19685 raft_consensus.cc:3037] T 604dbfa49eb747fb80692db93d6e5719 P d4653108eeb0438abdabf77a0f37ae26 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:27:39.076683 19170 raft_consensus.cc:3037] T 604dbfa49eb747fb80692db93d6e5719 P 01fa332a945a4f2589f1557320cdf416 [term 0 FOLLOWER]: Advancing to term 1
W20260811 22:27:39.097254 19261 tablet.cc:2369] T 604dbfa49eb747fb80692db93d6e5719 P 01fa332a945a4f2589f1557320cdf416: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:27:39.117458 19170 raft_consensus.cc:2454] T 604dbfa49eb747fb80692db93d6e5719 P 01fa332a945a4f2589f1557320cdf416 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c272d94669ef406185fc0d553c37685d in term 1.
I20260811 22:27:39.117478 19685 raft_consensus.cc:2454] T 604dbfa49eb747fb80692db93d6e5719 P d4653108eeb0438abdabf77a0f37ae26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c272d94669ef406185fc0d553c37685d in term 1.
I20260811 22:27:39.118085 19310 leader_election.cc:302] T 604dbfa49eb747fb80692db93d6e5719 P c272d94669ef406185fc0d553c37685d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 01fa332a945a4f2589f1557320cdf416, c272d94669ef406185fc0d553c37685d, d4653108eeb0438abdabf77a0f37ae26; no voters:
I20260811 22:27:39.118589 19829 raft_consensus.cc:2781] T 604dbfa49eb747fb80692db93d6e5719 P c272d94669ef406185fc0d553c37685d [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:27:39.119128 19829 raft_consensus.cc:686] T 604dbfa49eb747fb80692db93d6e5719 P c272d94669ef406185fc0d553c37685d [term 1 LEADER]: Becoming Leader. State: Replica: c272d94669ef406185fc0d553c37685d, State: Running, Role: LEADER
I20260811 22:27:39.119810 19829 consensus_queue.cc:237] T 604dbfa49eb747fb80692db93d6e5719 P c272d94669ef406185fc0d553c37685d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c272d94669ef406185fc0d553c37685d" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 33711 } } peers { permanent_uuid: "d4653108eeb0438abdabf77a0f37ae26" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 32869 } } peers { permanent_uuid: "01fa332a945a4f2589f1557320cdf416" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 45705 } }
I20260811 22:27:39.126441 18868 catalog_manager.cc:5495] T 604dbfa49eb747fb80692db93d6e5719 P c272d94669ef406185fc0d553c37685d reported cstate change: term changed from 0 to 1, leader changed from <none> to c272d94669ef406185fc0d553c37685d (127.29.106.2). New cstate: current_term: 1 leader_uuid: "c272d94669ef406185fc0d553c37685d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c272d94669ef406185fc0d553c37685d" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 33711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4653108eeb0438abdabf77a0f37ae26" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 32869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01fa332a945a4f2589f1557320cdf416" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 45705 } health_report { overall_health: UNKNOWN } } }
I20260811 22:27:39.167023 19800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:27:39.161221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45696 (local address 127.29.106.2:33711)
0811 22:27:39.161424 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0811 22:27:39.161430 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:27:39.161452 (+ 22us) server_negotiation.cc:407] Connection header received
0811 22:27:39.161684 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:27:39.161688 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:27:39.161778 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:27:39.161923 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:27:39.162489 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:27:39.163251 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:27:39.164820 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:27:39.165225 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:27:39.165333 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:27:39.165869 (+ 536us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:27:39.166458 (+ 589us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:27:39.166603 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0811 22:27:39.166694 (+ 91us) server_negotiation.cc:299] Negotiation successful
0811 22:27:39.166775 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260811 22:27:39.175987 19170 raft_consensus.cc:1261] T 604dbfa49eb747fb80692db93d6e5719 P 01fa332a945a4f2589f1557320cdf416 [term 1 FOLLOWER]: Refusing update from remote peer c272d94669ef406185fc0d553c37685d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:27:39.176357 19685 raft_consensus.cc:1261] T 604dbfa49eb747fb80692db93d6e5719 P d4653108eeb0438abdabf77a0f37ae26 [term 1 FOLLOWER]: Refusing update from remote peer c272d94669ef406185fc0d553c37685d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:27:39.177091 19838 consensus_queue.cc:1035] T 604dbfa49eb747fb80692db93d6e5719 P c272d94669ef406185fc0d553c37685d [LEADER]: Connected to new peer: Peer: permanent_uuid: "01fa332a945a4f2589f1557320cdf416" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 45705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260811 22:27:39.177922 19829 consensus_queue.cc:1035] T 604dbfa49eb747fb80692db93d6e5719 P c272d94669ef406185fc0d553c37685d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4653108eeb0438abdabf77a0f37ae26" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 32869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260811 22:27:39.179618 19525 tablet.cc:2369] T 604dbfa49eb747fb80692db93d6e5719 P c272d94669ef406185fc0d553c37685d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:27:39.219311 18869 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:27:39.219838 18869 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:27:39.225485 18869 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 57f2ba315f6a4571a60fbab368f571a7: Sending DeleteTablet for 3 replicas of tablet 604dbfa49eb747fb80692db93d6e5719
I20260811 22:27:39.227092 19389 tablet_service.cc:1514] Processing DeleteTablet for tablet 604dbfa49eb747fb80692db93d6e5719 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:27:39 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:45684
I20260811 22:27:39.227252 19643 tablet_service.cc:1514] Processing DeleteTablet for tablet 604dbfa49eb747fb80692db93d6e5719 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:27:39 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:39378
I20260811 22:27:39.227691 19136 tablet_service.cc:1514] Processing DeleteTablet for tablet 604dbfa49eb747fb80692db93d6e5719 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:27:39 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:42716
I20260811 22:27:39.230505 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 19011
I20260811 22:27:39.240321 19864 tablet_replica.cc:330] T 604dbfa49eb747fb80692db93d6e5719 P c272d94669ef406185fc0d553c37685d: stopping tablet replica
I20260811 22:27:39.240964 19864 raft_consensus.cc:2229] T 604dbfa49eb747fb80692db93d6e5719 P c272d94669ef406185fc0d553c37685d [term 1 LEADER]: Raft consensus shutting down.
I20260811 22:27:39.242197 19864 raft_consensus.cc:2258] T 604dbfa49eb747fb80692db93d6e5719 P c272d94669ef406185fc0d553c37685d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:27:39.246083 19864 ts_tablet_manager.cc:1905] T 604dbfa49eb747fb80692db93d6e5719 P c272d94669ef406185fc0d553c37685d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:27:39.246606 19865 tablet_replica.cc:330] T 604dbfa49eb747fb80692db93d6e5719 P d4653108eeb0438abdabf77a0f37ae26: stopping tablet replica
I20260811 22:27:39.247018 19865 raft_consensus.cc:2229] T 604dbfa49eb747fb80692db93d6e5719 P d4653108eeb0438abdabf77a0f37ae26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:27:39.250507 19865 raft_consensus.cc:2258] T 604dbfa49eb747fb80692db93d6e5719 P d4653108eeb0438abdabf77a0f37ae26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:27:39.251578 19866 tablet_replica.cc:330] T 604dbfa49eb747fb80692db93d6e5719 P 01fa332a945a4f2589f1557320cdf416: stopping tablet replica
I20260811 22:27:39.252063 19866 raft_consensus.cc:2229] T 604dbfa49eb747fb80692db93d6e5719 P 01fa332a945a4f2589f1557320cdf416 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:27:39.252753 19866 raft_consensus.cc:2258] T 604dbfa49eb747fb80692db93d6e5719 P 01fa332a945a4f2589f1557320cdf416 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:27:39.253007 19865 ts_tablet_manager.cc:1905] T 604dbfa49eb747fb80692db93d6e5719 P d4653108eeb0438abdabf77a0f37ae26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:27:39.255494 19866 ts_tablet_manager.cc:1905] T 604dbfa49eb747fb80692db93d6e5719 P 01fa332a945a4f2589f1557320cdf416: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 11 22:27:39 dist-test-slave-gjrz krb5kdc[17968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.1@KRBTEST.COM
I20260811 22:27:39.269833 19807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:27:39.231950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42730 (local address 127.29.106.1:45705)
0811 22:27:39.232147 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0811 22:27:39.232154 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:27:39.243464 (+ 11310us) server_negotiation.cc:407] Connection header received
0811 22:27:39.243763 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:27:39.243770 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:27:39.243919 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:27:39.244000 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:27:39.247615 (+ 3615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:27:39.249158 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:27:39.251915 (+ 2757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:27:39.253229 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:27:39.253351 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:27:39.259095 (+ 5744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:27:39.259111 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:27:39.259130 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:27:39.259171 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:27:39.265041 (+ 5870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:27:39.265758 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:27:39.265767 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:27:39.265770 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:27:39.266030 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:27:39.266734 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:27:39.266740 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:27:39.266743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:27:39.267893 (+ 1150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:27:39.268416 (+ 523us) server_negotiation.cc:1035] Waiting for connection context
0811 22:27:39.268641 (+ 225us) server_negotiation.cc:299] Negotiation successful
0811 22:27:39.268930 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260811 22:27:39.271137 19240 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:27:39.305049 19864 ts_tablet_manager.cc:1918] T 604dbfa49eb747fb80692db93d6e5719 P c272d94669ef406185fc0d553c37685d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:27:39.305270 19864 log.cc:1196] T 604dbfa49eb747fb80692db93d6e5719 P c272d94669ef406185fc0d553c37685d: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-1/wal/wals/604dbfa49eb747fb80692db93d6e5719
I20260811 22:27:39.305902 19864 ts_tablet_manager.cc:1939] T 604dbfa49eb747fb80692db93d6e5719 P c272d94669ef406185fc0d553c37685d: Deleting consensus metadata
I20260811 22:27:39.365495 18847 catalog_manager.cc:4841] TS c272d94669ef406185fc0d553c37685d (127.29.106.2:33711): tablet 604dbfa49eb747fb80692db93d6e5719 (table test-table [id=9e81ec0393824482aec52300e67f80ee]) successfully deleted
I20260811 22:27:39.393258 19865 ts_tablet_manager.cc:1918] T 604dbfa49eb747fb80692db93d6e5719 P d4653108eeb0438abdabf77a0f37ae26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:27:39.393473 19865 log.cc:1196] T 604dbfa49eb747fb80692db93d6e5719 P d4653108eeb0438abdabf77a0f37ae26: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786487105757681-30120-0/minicluster-data/ts-2/wal/wals/604dbfa49eb747fb80692db93d6e5719
I20260811 22:27:39.393976 19865 ts_tablet_manager.cc:1939] T 604dbfa49eb747fb80692db93d6e5719 P d4653108eeb0438abdabf77a0f37ae26: Deleting consensus metadata
I20260811 22:27:39.397396 18848 catalog_manager.cc:4841] TS d4653108eeb0438abdabf77a0f37ae26 (127.29.106.3:32869): tablet 604dbfa49eb747fb80692db93d6e5719 (table test-table [id=9e81ec0393824482aec52300e67f80ee]) successfully deleted
I20260811 22:27:39.422477 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 19011
I20260811 22:27:39.444257 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 19271
W20260811 22:27:39.444217 18845 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.106.1:45705 (error 108)
I20260811 22:27:39.446509 19873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:27:39.445651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:45705 (local address 127.0.0.1:42742)
0811 22:27:39.446112 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0811 22:27:39.446319 (+ 207us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.1:45705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":206,"threads_started":1}
W20260811 22:27:39.446780 18845 catalog_manager.cc:4584] TS 01fa332a945a4f2589f1557320cdf416 (127.29.106.1:45705): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 604dbfa49eb747fb80692db93d6e5719: Network error: Client connection negotiation failed: client connection to 127.29.106.1:45705: connect: Connection refused (error 111)
Aug 11 22:27:39 dist-test-slave-gjrz krb5kdc[17968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.2@KRBTEST.COM
I20260811 22:27:39.462843 19800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:27:39.445095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45710 (local address 127.29.106.2:33711)
0811 22:27:39.445291 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0811 22:27:39.445297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:27:39.445423 (+ 126us) server_negotiation.cc:407] Connection header received
0811 22:27:39.445665 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:27:39.445670 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:27:39.445773 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:27:39.445883 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:27:39.447798 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:27:39.448674 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:27:39.450828 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:27:39.451440 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:27:39.451539 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:27:39.456525 (+ 4986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:27:39.456539 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:27:39.456555 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:27:39.456592 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:27:39.459571 (+ 2979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:27:39.460215 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:27:39.460223 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:27:39.460231 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:27:39.460424 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:27:39.460792 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:27:39.460797 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:27:39.460801 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:27:39.461163 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:27:39.461455 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0811 22:27:39.461780 (+ 325us) server_negotiation.cc:299] Negotiation successful
0811 22:27:39.462573 (+ 793us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260811 22:27:39.463634 19510 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:27:39.500416 19873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:27:39.499893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:45705 (local address 127.0.0.1:42750)
0811 22:27:39.500139 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0811 22:27:39.500238 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.1:45705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260811 22:27:39.501515 19873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:27:39.501120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:45705 (local address 127.0.0.1:42758)
0811 22:27:39.501258 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0811 22:27:39.501361 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.1:45705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260811 22:27:39.562660 19873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:27:39.561526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:45705 (local address 127.0.0.1:42766)
0811 22:27:39.561757 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0811 22:27:39.562459 (+ 702us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.1:45705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260811 22:27:39.563719 19873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:27:39.563350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:45705 (local address 127.0.0.1:42778)
0811 22:27:39.563493 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0811 22:27:39.563580 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.1:45705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260811 22:27:39.577203 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 19271
I20260811 22:27:39.603113 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 19531
Aug 11 22:27:39 dist-test-slave-gjrz krb5kdc[17968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.3@KRBTEST.COM
I20260811 22:27:39.617334 19878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:27:39.603798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39388 (local address 127.29.106.3:32869)
0811 22:27:39.604224 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0811 22:27:39.604229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:27:39.604246 (+ 17us) server_negotiation.cc:407] Connection header received
0811 22:27:39.604303 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:27:39.604306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:27:39.604374 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:27:39.604463 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:27:39.605647 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:27:39.606341 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:27:39.607606 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:27:39.607989 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:27:39.608088 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:27:39.612350 (+ 4262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:27:39.612364 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:27:39.612381 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:27:39.612416 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:27:39.614960 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:27:39.615685 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:27:39.615692 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:27:39.615694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:27:39.615823 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:27:39.616140 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:27:39.616143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:27:39.616145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:27:39.616490 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:27:39.616710 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0811 22:27:39.616896 (+ 186us) server_negotiation.cc:299] Negotiation successful
0811 22:27:39.617068 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":219,"threads_started":1}
I20260811 22:27:39.617928 19768 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:27:39.644070 19873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:27:39.643651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:45705 (local address 127.0.0.1:42794)
0811 22:27:39.643856 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0811 22:27:39.643938 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.1:45705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260811 22:27:39.645027 19873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:27:39.644667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:45705 (local address 127.0.0.1:42800)
0811 22:27:39.644829 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0811 22:27:39.644901 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.1:45705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260811 22:27:39.706015 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 19531
I20260811 22:27:39.728940 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 18818
I20260811 22:27:39.730055 18929 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:27:39.842038 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 18818
I20260811 22:27:39.842080 19873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:27:39.841063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:45705 (local address 127.0.0.1:36980)
0811 22:27:39.841807 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0811 22:27:39.841933 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.1:45705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":606}
I20260811 22:27:39.874631 30120 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260811 22:27:40.218461 30120 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260811 22:27:40.218770 30120 mini_ranger.cc:66] Stopping Ranger...
I20260811 22:27:40.562809 30120 mini_ranger.cc:68] Stopped Ranger
2026-08-11 22:27:40.563 UTC [18027] LOG: received smart shutdown request
2026-08-11 22:27:40.567 UTC [18027] LOG: background worker "logical replication launcher" (PID 18048) exited with exit code 1
2026-08-11 22:27:40.567 UTC [18042] LOG: shutting down
2026-08-11 22:27:40.676 UTC [18027] LOG: database system is shut down
2026-08-11T22:27:40Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (51495 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:27:40 dist-test-slave-gjrz krb5kdc[19896](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:27:40 dist-test-slave-gjrz krb5kdc[19896](info): set up 2 sockets
Aug 11 22:27:40 dist-test-slave-gjrz krb5kdc[19896](info): commencing operation
krb5kdc: starting...
W20260811 22:27:42.067164 30120 mini_kdc.cc:121] Time spent starting KDC: real 1.237s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Aug 11 22:27:42 dist-test-slave-gjrz krb5kdc[19896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:27:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:27:42Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.29.106.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.29.106.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.29.106.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/rangeradmin_127.29.106.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.29.106.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/rangeradmin_127.29.106.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.29.106.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.29.106.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.29.106.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/rangerlookup_127.29.106.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.29.106.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/rangerlookup_127.29.106.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.29.106.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.29.106.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.29.106.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/HTTP_127.29.106.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.29.106.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/HTTP_127.29.106.20@KRBTEST.COM.keytab.
I20260811 22:27:42.388259 30120 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task7tIZk9/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/postgres -l logfile start
2026-08-11 22:27:44.658 UTC [19951] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-11 22:27:44.658 UTC [19951] LOG: listening on IPv4 address "127.29.106.20", port 45559
2026-08-11 22:27:44.661 UTC [19951] LOG: listening on Unix socket "/tmp/.s.PGSQL.45559"
2026-08-11 22:27:44.682 UTC [19956] LOG: database system was shut down at 2026-08-11 22:27:43 UTC
2026-08-11 22:27:44.685 UTC [19951] LOG: database system is ready to accept connections
I20260811 22:27:45.815557 30120 mini_postgres.cc:96] Postgres bound to 45559
2026-08-11 22:27:45.837 UTC [19967] FATAL: database "slave" does not exist
127.29.106.20:45559 - accepting connections
I20260811 22:27:45.838568 30120 mini_ranger.cc:161] Starting Ranger...
I20260811 22:27:45.875427 30120 mini_ranger.cc:84] Created miniranger Postgres user
I20260811 22:27:46.298986 30120 mini_ranger.cc:87] Created ranger Postgres database
I20260811 22:27:46.299101 30120 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ranger-admin
2026-08-11 22:27:46,577 [I] DB FLAVOR :POSTGRES
2026-08-11 22:27:46,577 [I] --------- Verifying Ranger DB connection ---------
2026-08-11 22:27:46,578 [I] Checking connection..
2026-08-11 22:27:46,578 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7tIZk9/build/asan/bin/postgresql.jar:/tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.106.20:45559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-11 22:27:46,818 [I] Checking connection passed.
2026-08-11 22:27:46,818 [I] --------- Verifying version history table ---------
2026-08-11 22:27:46,818 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7tIZk9/build/asan/bin/postgresql.jar:/tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.106.20:45559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-11 22:27:47,073 [I] Table x_db_version_h does not exist in database ranger
2026-08-11 22:27:47,073 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7tIZk9/build/asan/bin/postgresql.jar:/tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.106.20:45559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-11 22:27:47,315 [I] Table x_db_version_h does not exist in database ranger
2026-08-11 22:27:47,316 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-11 22:27:47,316 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7tIZk9/build/asan/bin/postgresql.jar:/tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.106.20:45559/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-11 22:27:47.566 UTC [20072] WARNING: there is no transaction in progress
2026-08-11 22:27:47,574 [I] create_dbversion_catalog.sql file imported successfully
2026-08-11 22:27:47,575 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7tIZk9/build/asan/bin/postgresql.jar:/tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.106.20:45559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-11 22:27:47,831 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-11 22:27:47,831 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-11 22:27:47,831 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7tIZk9/build/asan/bin/postgresql.jar:/tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.106.20:45559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-11 22:27:48,071 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7tIZk9/build/asan/bin/postgresql.jar:/tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.106.20:45559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-11 22:27:48,326 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7tIZk9/build/asan/bin/postgresql.jar:/tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.106.20:45559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-gjrz.c.gcp-upstream.internal','N') ;"
2026-08-11 22:27:48,565 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7tIZk9/build/asan/bin/postgresql.jar:/tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.106.20:45559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-11 22:27:48,822 [I] Table x_portal_user does not exist in database ranger
2026-08-11 22:27:48,822 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7tIZk9/build/asan/bin/postgresql.jar:/tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.106.20:45559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-11 22:27:49,079 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-11 22:27:49,079 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7tIZk9/build/asan/bin/postgresql.jar:/tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.106.20:45559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-11 22:27:49,331 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7tIZk9/build/asan/bin/postgresql.jar:/tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.106.20:45559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-11 22:27:49,563 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-11 22:27:49,563 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7tIZk9/build/asan/bin/postgresql.jar:/tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.106.20:45559/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-11 22:27:50.455 UTC [20261] WARNING: there is no transaction in progress
2026-08-11 22:27:50.470 UTC [20261] WARNING: there is no transaction in progress
2026-08-11 22:27:50.485 UTC [20261] WARNING: there is no transaction in progress
2026-08-11 22:27:50.499 UTC [20261] WARNING: there is no transaction in progress
2026-08-11 22:27:50.516 UTC [20261] WARNING: there is no transaction in progress
2026-08-11 22:27:50.531 UTC [20261] WARNING: there is no transaction in progress
2026-08-11 22:27:50.580 UTC [20261] WARNING: there is no transaction in progress
2026-08-11 22:27:50.585 UTC [20261] WARNING: there is no transaction in progress
2026-08-11 22:27:50.598 UTC [20261] WARNING: there is no transaction in progress
2026-08-11 22:27:50.607 UTC [20261] WARNING: there is no transaction in progress
2026-08-11 22:27:50.616 UTC [20261] WARNING: there is no transaction in progress
2026-08-11 22:27:50.628 UTC [20261] WARNING: there is no transaction in progress
2026-08-11 22:27:50.637 UTC [20261] WARNING: there is no transaction in progress
2026-08-11 22:27:50.643 UTC [20261] WARNING: there is no transaction in progress
2026-08-11 22:27:50.648 UTC [20261] WARNING: there is no transaction in progress
2026-08-11 22:27:50.654 UTC [20261] WARNING: there is no transaction in progress
2026-08-11 22:27:51,216 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-11 22:27:51,216 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7tIZk9/build/asan/bin/postgresql.jar:/tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.106.20:45559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-11 22:27:51,454 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-11 22:27:51,454 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7tIZk9/build/asan/bin/postgresql.jar:/tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.106.20:45559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-11 22:27:51,699 [I] Table x_portal_user already exists in database 'ranger'
2026-08-11 22:27:51,699 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7tIZk9/build/asan/bin/postgresql.jar:/tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.106.20:45559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-11 22:27:51,951 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-11 22:27:51,951 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7tIZk9/build/asan/bin/postgresql.jar:/tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.106.20:45559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-11 22:27:52,211 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7tIZk9/build/asan/bin/postgresql.jar:/tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.106.20:45559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-11 22:27:52,463 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7tIZk9/build/asan/bin/postgresql.jar:/tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.106.20:45559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-gjrz.c.gcp-upstream.internal';"
2026-08-11 22:27:52,721 [I] CORE_DB_SCHEMA import status has been updated
2026-08-11 22:27:52,721 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7tIZk9/build/asan/bin/postgresql.jar:/tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.106.20:45559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-11 22:27:52,977 [I] DB_PATCHES have already been applied
I20260811 22:27:52.983728 30120 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ranger-admin:/tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task7tIZk9/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task7tIZk9/build/asan/bin/postgresql.jar:/tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260811 22:27:52.983807 30120 mini_ranger.cc:193] Using host: 127.29.106.20
I20260811 22:27:53.002897 30120 mini_ranger.cc:227] Ranger admin URL: http://127.29.106.20:39093
Aug 11, 2026 10:27:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 11, 2026 10:27:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 11, 2026 10:27:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ranger-admin/ews/webapp
Aug 11, 2026 10:27:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 11, 2026 10:27:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 11, 2026 10:27:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ranger-admin/ews/webapp].
Aug 11, 2026 10:27:53 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260811 22:27:54.343187 30120 mini_ranger.cc:160] Time spent starting Ranger: real 8.505s user 0.009s sys 0.046s
Aug 11, 2026 10:28:12 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 11, 2026 10:28:12 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Aug 11, 2026 10:28:12 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 11, 2026 10:28:12 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 11, 2026 10:28:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 11, 2026 10:28:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 11, 2026 10:28:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 11, 2026 10:28:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 11, 2026 10:28:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 11, 2026 10:28:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 11, 2026 10:28:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 11, 2026 10:28:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 11, 2026 10:28:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 11, 2026 10:28:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 11, 2026 10:28:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 11, 2026 10:28:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 11, 2026 10:28:12 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 11, 2026 10:28:13 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260811 22:28:14.246912 30120 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.29.106.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.29.106.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.29.106.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/rangerkms_127.29.106.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.29.106.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/rangerkms_127.29.106.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.29.106.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.29.106.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.29.106.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/HTTP_127.29.106.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.29.106.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/HTTP_127.29.106.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 11 22:28:14 dist-test-slave-gjrz krb5kdc[19896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487294, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260811 22:28:14.417070 30120 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260811 22:28:14.438251 30120 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260811 22:28:15.165459 30120 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260811 22:28:15.165581 30120 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ranger-kms
I20260811 22:28:15.165619 30120 mini_ranger_kms.cc:227] Using postgres at 127.29.106.20:45559
2026-08-11 22:28:15,511 [I] DB FLAVOR :POSTGRES
2026-08-11 22:28:15,512 [I] --------- Verifying Ranger DB connection ---------
2026-08-11 22:28:15,512 [I] Checking connection
2026-08-11 22:28:15,512 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7tIZk9/build/asan/bin/postgresql.jar:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.106.20:45559/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-11 22:28:15,747 [I] connection success
2026-08-11 22:28:15,747 [I] --------- Verifying Ranger DB tables ---------
2026-08-11 22:28:15,747 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-11 22:28:15,747 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7tIZk9/build/asan/bin/postgresql.jar:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.106.20:45559/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-11 22:28:15,976 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-11 22:28:15,977 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-11 22:28:15,977 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-11 22:28:15,977 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7tIZk9/build/asan/bin/postgresql.jar:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.106.20:45559/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-11 22:28:16,261 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260811 22:28:16.700843 30120 mini_ranger_kms.cc:310] Created kms service
I20260811 22:28:16.908310 30120 mini_ranger_kms.cc:326] Created kudu user
I20260811 22:28:16.952776 30120 mini_ranger_kms.cc:343] Created rangerkms user
Aug 11 22:28:16 dist-test-slave-gjrz krb5kdc[19896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487294, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.29.106.20@KRBTEST.COM
I20260811 22:28:17.314074 30120 mini_ranger_kms.cc:384] Added ranger policy
I20260811 22:28:17.314400 30120 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ranger-kms:/tmp/dist-test-task7tIZk9/build/asan/bin/postgresql.jar:/tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task7tIZk9/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260811 22:28:17.314464 30120 mini_ranger_kms.cc:240] Using host: 127.29.106.20
I20260811 22:28:17.334952 30120 mini_ranger_kms.cc:276] Ranger KMS PID: 20602
I20260811 22:28:17.335078 30120 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.29.106.20:58037
Aug 11, 2026 10:28:18 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 11, 2026 10:28:18 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 11, 2026 10:28:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 11, 2026 10:28:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 11, 2026 10:28:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ranger-kms/ews/webapp].
Aug 11, 2026 10:28:18 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 11 22:28:19 dist-test-slave-gjrz krb5kdc[19896](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786487299, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.29.106.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:28:19.984987 30120 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.568s user 0.037s sys 0.332s
I20260811 22:28:19.985210 30120 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260811 22:28:19.985371 30120 mini_ranger_kms.cc:401] 127.29.106.20:58037/kms/v1/keys
Aug 11 22:28:20 dist-test-slave-gjrz krb5kdc[19896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487299, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.29.106.20@KRBTEST.COM for HTTP/127.29.106.20@KRBTEST.COM
Aug 11 22:28:20 dist-test-slave-gjrz krb5kdc[19896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487299, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.29.106.20@KRBTEST.COM for HTTP/127.29.106.20@KRBTEST.COM
Aug 11 22:28:22 dist-test-slave-gjrz krb5kdc[19896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.29.106.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
I20260811 22:28:22.608589 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:36269
--webserver_interface=127.29.106.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:41611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:36269
--ranger_config_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.106.20:58037/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:28:22.931895 20700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:28:22.932168 20700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:28:22.932286 20700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:28:22.939234 20700 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:28:22.939347 20700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:28:22.939476 20700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:28:22.939545 20700 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:28:22.939611 20700 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:28:22.949272 20700 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:41611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/master-0/wal
--ranger_kms_url=127.29.106.20:58037/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:36269
--ranger_config_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:36269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.106.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:28:22.950462 20700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:28:22.951746 20700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:28:22.963132 20709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:28:22.963169 20713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:28:22.963210 20708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:28:22.964598 20700 server_base.cc:1034] running on GCE node
I20260811 22:28:22.965322 20700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:28:22.966629 20700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:28:22.967832 20700 hybrid_clock.cc:648] HybridClock initialized: now 1786487302967764 us; error 43 us; skew 500 ppm
I20260811 22:28:22.968215 20700 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:28:22 dist-test-slave-gjrz krb5kdc[19896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487302, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:28:22.976603 20700 init.cc:377] Logged in from keytab as kudu/127.29.106.62@KRBTEST.COM (short username kudu)
I20260811 22:28:22.977950 20700 webserver.cc:460] Webserver started at http://127.29.106.62:40715/ using document root <none> and password file <none>
I20260811 22:28:22.978528 20700 fs_manager.cc:359] Metadata directory not provided
I20260811 22:28:22.978639 20700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:28:22.978925 20700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 22:28:22 dist-test-slave-gjrz krb5kdc[19896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487302, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for HTTP/127.29.106.20@KRBTEST.COM
I20260811 22:28:23.036190 20700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "dbc85509642145079cb9f19ed04547ff"
format_stamp: "Formatted at 2026-08-11 22:28:23 on dist-test-slave-gjrz"
server_key: "94d3288abd16f9ded9b874bdf2232501"
server_key_iv: "04e6035cabdc0c07125b012057449441"
server_key_version: "kuduclusterkey@0"
I20260811 22:28:23.037258 20700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "dbc85509642145079cb9f19ed04547ff"
format_stamp: "Formatted at 2026-08-11 22:28:23 on dist-test-slave-gjrz"
server_key: "94d3288abd16f9ded9b874bdf2232501"
server_key_iv: "04e6035cabdc0c07125b012057449441"
server_key_version: "kuduclusterkey@0"
I20260811 22:28:23.044253 20700 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260811 22:28:23.063391 20725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:28:23.064854 20700 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260811 22:28:23.065032 20700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "dbc85509642145079cb9f19ed04547ff"
format_stamp: "Formatted at 2026-08-11 22:28:23 on dist-test-slave-gjrz"
server_key: "94d3288abd16f9ded9b874bdf2232501"
server_key_iv: "04e6035cabdc0c07125b012057449441"
server_key_version: "kuduclusterkey@0"
I20260811 22:28:23.065233 20700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:28:23.084271 20700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:28:23.087690 20700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:28:23.087978 20700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:28:23.104822 20700 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.62:36269
I20260811 22:28:23.104843 20826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.62:36269 every 8 connection(s)
I20260811 22:28:23.106098 20700 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:28:23.109656 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 20700
I20260811 22:28:23.109844 20827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:28:23.109864 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:28:23.110718 30120 external_mini_cluster.cc:1428] Setting key bef902a0973cd3f4f3925e97d8090f2b
I20260811 22:28:23.121753 20827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbc85509642145079cb9f19ed04547ff: Bootstrap starting.
Aug 11 22:28:23 dist-test-slave-gjrz krb5kdc[19896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:28:23.125074 20827 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbc85509642145079cb9f19ed04547ff: Neither blocks nor log segments found. Creating new log.
I20260811 22:28:23.126067 20827 log.cc:824] T 00000000000000000000000000000000 P dbc85509642145079cb9f19ed04547ff: Log is configured to *not* fsync() on all Append() calls
I20260811 22:28:23.129024 20827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbc85509642145079cb9f19ed04547ff: No bootstrap required, opened a new log
I20260811 22:28:23.132718 20833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:23.112855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54570 (local address 127.29.106.62:36269)
0811 22:28:23.113611 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:23.113623 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:23.113666 (+ 43us) server_negotiation.cc:407] Connection header received
0811 22:28:23.115994 (+ 2328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:23.116022 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:23.116423 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:23.116783 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:28:23.118151 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:23.119523 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:23.120835 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:23.121496 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:23.121655 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:23.125694 (+ 4039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:28:23.125702 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:28:23.125716 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:28:23.125756 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:28:23.128932 (+ 3176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:23.129597 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:23.129604 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:23.129610 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:23.129789 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:23.130144 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:23.130150 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:23.130153 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:23.130650 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:28:23.130934 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:23.131429 (+ 495us) server_negotiation.cc:299] Negotiation successful
0811 22:28:23.131797 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":342,"threads_started":1}
I20260811 22:28:23.137380 20827 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbc85509642145079cb9f19ed04547ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbc85509642145079cb9f19ed04547ff" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 36269 } }
I20260811 22:28:23.137701 20827 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbc85509642145079cb9f19ed04547ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:28:23.137790 20827 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbc85509642145079cb9f19ed04547ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbc85509642145079cb9f19ed04547ff, State: Initialized, Role: FOLLOWER
I20260811 22:28:23.138296 20827 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbc85509642145079cb9f19ed04547ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbc85509642145079cb9f19ed04547ff" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 36269 } }
I20260811 22:28:23.138506 20827 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbc85509642145079cb9f19ed04547ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:28:23.138633 20827 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbc85509642145079cb9f19ed04547ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:28:23.138782 20827 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbc85509642145079cb9f19ed04547ff [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:28:23.141129 20827 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbc85509642145079cb9f19ed04547ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbc85509642145079cb9f19ed04547ff" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 36269 } }
I20260811 22:28:23.141639 20827 leader_election.cc:302] T 00000000000000000000000000000000 P dbc85509642145079cb9f19ed04547ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbc85509642145079cb9f19ed04547ff; no voters:
I20260811 22:28:23.142172 20827 leader_election.cc:288] T 00000000000000000000000000000000 P dbc85509642145079cb9f19ed04547ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:28:23.142270 20836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbc85509642145079cb9f19ed04547ff [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:28:23.142889 20836 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbc85509642145079cb9f19ed04547ff [term 1 LEADER]: Becoming Leader. State: Replica: dbc85509642145079cb9f19ed04547ff, State: Running, Role: LEADER
I20260811 22:28:23.143445 20836 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbc85509642145079cb9f19ed04547ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbc85509642145079cb9f19ed04547ff" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 36269 } }
I20260811 22:28:23.144043 20827 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbc85509642145079cb9f19ed04547ff [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:28:23.146308 20839 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbc85509642145079cb9f19ed04547ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbc85509642145079cb9f19ed04547ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbc85509642145079cb9f19ed04547ff" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 36269 } } }
I20260811 22:28:23.146705 20839 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbc85509642145079cb9f19ed04547ff [sys.catalog]: This master's current role is: LEADER
I20260811 22:28:23.146437 20840 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbc85509642145079cb9f19ed04547ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbc85509642145079cb9f19ed04547ff. Latest consensus state: current_term: 1 leader_uuid: "dbc85509642145079cb9f19ed04547ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbc85509642145079cb9f19ed04547ff" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 36269 } } }
I20260811 22:28:23.146998 20840 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbc85509642145079cb9f19ed04547ff [sys.catalog]: This master's current role is: LEADER
I20260811 22:28:23.152271 20851 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:28:23.152668 20827 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260811 22:28:23.165323 20851 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:28:23.174316 20851 catalog_manager.cc:1347] Generated new cluster ID: 0cccbdbdee424760a73815c776a42016
I20260811 22:28:23.174422 20851 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:28:23.185302 20851 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:28:23.187126 20851 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:28:23.207975 20851 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbc85509642145079cb9f19ed04547ff: Generated new TSK 0
I20260811 22:28:23.208761 20851 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.106.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab.
Entry for principal kudu/127.29.106.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab.
I20260811 22:28:23.946413 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.106.20:58037/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.1:0
--local_ip_for_outbound_sockets=127.29.106.1
--webserver_interface=127.29.106.1
--webserver_port=0
--tserver_master_addrs=127.29.106.62:36269
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab
--principal=kudu/127.29.106.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:41611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 22:28:24 dist-test-slave-gjrz krb5kdc[19896](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786487304, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260811 22:28:24.373624 20912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:28:24.373998 20912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:28:24.374166 20912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:28:24.385331 20912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:28:24.385468 20912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:28:24.385658 20912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.1
I20260811 22:28:24.400687 20912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:41611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.29.106.20:58037/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab
--principal=kudu/127.29.106.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.106.1
--webserver_port=0
--tserver_master_addrs=127.29.106.62:36269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.1
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:28:24.402395 20912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:28:24.404175 20912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:28:24.422834 20931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:28:24.422981 20935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:28:24.423476 20938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:28:24.423900 20912 server_base.cc:1034] running on GCE node
I20260811 22:28:24.424623 20912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:28:24.425836 20912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:28:24.427129 20912 hybrid_clock.cc:648] HybridClock initialized: now 1786487304427064 us; error 48 us; skew 500 ppm
I20260811 22:28:24.427539 20912 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:28:24 dist-test-slave-gjrz krb5kdc[19896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487304, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:28:24.431684 20912 init.cc:377] Logged in from keytab as kudu/127.29.106.1@KRBTEST.COM (short username kudu)
I20260811 22:28:24.442942 20912 webserver.cc:460] Webserver started at http://127.29.106.1:44053/ using document root <none> and password file <none>
I20260811 22:28:24.443847 20912 fs_manager.cc:359] Metadata directory not provided
I20260811 22:28:24.444038 20912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:28:24.444417 20912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 22:28:24 dist-test-slave-gjrz krb5kdc[19896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487304, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.1@KRBTEST.COM for HTTP/127.29.106.20@KRBTEST.COM
I20260811 22:28:24.461598 20912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-0/data/instance:
uuid: "05a49b25fe69430aa8303b2c5659158d"
format_stamp: "Formatted at 2026-08-11 22:28:24 on dist-test-slave-gjrz"
server_key: "d13343c28432478b5fdac0bfcb4d42ae"
server_key_iv: "4bea3e1e50c4447fef23a5cce0517b34"
server_key_version: "kuduclusterkey@0"
I20260811 22:28:24.462642 20912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance:
uuid: "05a49b25fe69430aa8303b2c5659158d"
format_stamp: "Formatted at 2026-08-11 22:28:24 on dist-test-slave-gjrz"
server_key: "d13343c28432478b5fdac0bfcb4d42ae"
server_key_iv: "4bea3e1e50c4447fef23a5cce0517b34"
server_key_version: "kuduclusterkey@0"
I20260811 22:28:24.474068 20912 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.009s sys 0.004s
I20260811 22:28:24.516404 20951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:28:24.518776 20912 fs_manager.cc:725] Time spent opening block manager: real 0.028s user 0.014s sys 0.013s
I20260811 22:28:24.519062 20912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-0/wal
uuid: "05a49b25fe69430aa8303b2c5659158d"
format_stamp: "Formatted at 2026-08-11 22:28:24 on dist-test-slave-gjrz"
server_key: "d13343c28432478b5fdac0bfcb4d42ae"
server_key_iv: "4bea3e1e50c4447fef23a5cce0517b34"
server_key_version: "kuduclusterkey@0"
I20260811 22:28:24.519451 20912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:28:24.540774 20912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:28:24.545742 20912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:28:24.546094 20912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:28:24.547325 20912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:28:24.549122 20912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:28:24.549225 20912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:28:24.549360 20912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:28:24.549424 20912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:28:24.624742 20912 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.1:36207
I20260811 22:28:24.624776 21163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.1:36207 every 8 connection(s)
I20260811 22:28:24.626405 20912 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
I20260811 22:28:24.633265 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 20912
I20260811 22:28:24.633467 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance
I20260811 22:28:24.634320 30120 external_mini_cluster.cc:1428] Setting key fb1969e8ae186da175f0ea95e1676884
Aug 11 22:28:24 dist-test-slave-gjrz krb5kdc[19896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487304, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.1@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:28:24.683027 21168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:24.629388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:34831 (local address 127.29.106.62:36269)
0811 22:28:24.659822 (+ 30434us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:24.659831 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:24.663326 (+ 3495us) server_negotiation.cc:407] Connection header received
0811 22:28:24.664793 (+ 1467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:24.664799 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:24.664902 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:24.665011 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:28:24.667018 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:24.668127 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:24.670842 (+ 2715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:24.671483 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:24.671593 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:24.675960 (+ 4367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:28:24.675966 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:28:24.675970 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:28:24.676018 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:28:24.679312 (+ 3294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:24.680206 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:24.680212 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:24.680215 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:24.680374 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:24.680901 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:24.680906 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:24.680910 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:24.681283 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:28:24.681410 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:24.682434 (+ 1024us) server_negotiation.cc:299] Negotiation successful
0811 22:28:24.682657 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30277,"thread_start_us":30428,"threads_started":1}
WARNING: no policy specified for kudu/127.29.106.2@KRBTEST.COM; defaulting to no policy
I20260811 22:28:24.685487 21170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:24.629984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:36269 (local address 127.29.106.1:34831)
0811 22:28:24.660529 (+ 30545us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:24.660603 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0811 22:28:24.663056 (+ 2453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:28:24.665400 (+ 2344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:28:24.665412 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:28:24.666028 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:28:24.666818 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:24.666844 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:24.668522 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:24.668528 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:24.670668 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:24.670700 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:24.671891 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:24.671898 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:24.672195 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:24.672981 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:28:24.673007 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:28:24.675715 (+ 2708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:28:24.679464 (+ 3749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:28:24.679473 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:28:24.679488 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:28:24.680060 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:28:24.680523 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:28:24.680527 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:28:24.680530 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:28:24.680764 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:28:24.681531 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:28:24.681539 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:28:24.681982 (+ 443us) client_negotiation.cc:769] Sending connection context
0811 22:28:24.683679 (+ 1697us) client_negotiation.cc:240] Negotiation successful
0811 22:28:24.684229 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30196,"thread_start_us":30953,"threads_started":1}
I20260811 22:28:24.687141 21164 heartbeater.cc:347] Connected to a master server at 127.29.106.62:36269
I20260811 22:28:24.687456 21164 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.2@KRBTEST.COM" created.
I20260811 22:28:24.688297 21164 heartbeater.cc:511] Master 127.29.106.62:36269 requested a full tablet report, sending...
I20260811 22:28:24.690696 20757 ts_manager.cc:194] Registered new tserver with Master: 05a49b25fe69430aa8303b2c5659158d (127.29.106.1:36207)
I20260811 22:28:24.692628 20757 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.1@KRBTEST.COM'} at 127.29.106.1:34831
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab.
Entry for principal kudu/127.29.106.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab.
I20260811 22:28:24.725055 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.106.20:58037/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.2:0
--local_ip_for_outbound_sockets=127.29.106.2
--webserver_interface=127.29.106.2
--webserver_port=0
--tserver_master_addrs=127.29.106.62:36269
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab
--principal=kudu/127.29.106.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:41611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 22:28:24 dist-test-slave-gjrz krb5kdc[19896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487304, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for HTTP/127.29.106.20@KRBTEST.COM
Aug 11 22:28:25 dist-test-slave-gjrz krb5kdc[19896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487304, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for HTTP/127.29.106.20@KRBTEST.COM
W20260811 22:28:25.224611 21174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:28:25.225148 21174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:28:25.225370 21174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:28:25.236747 21174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:28:25.236958 21174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:28:25.237195 21174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.2
I20260811 22:28:25.252684 21174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:41611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.29.106.20:58037/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab
--principal=kudu/127.29.106.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.106.2
--webserver_port=0
--tserver_master_addrs=127.29.106.62:36269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.2
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:28:25.254696 21174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:28:25.256578 21174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:28:25.301452 21186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:28:25.303699 21187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:28:25.303958 21193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:28:25.305480 21174 server_base.cc:1034] running on GCE node
I20260811 22:28:25.306442 21174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:28:25.312372 21174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:28:25.313673 21174 hybrid_clock.cc:648] HybridClock initialized: now 1786487305313596 us; error 59 us; skew 500 ppm
I20260811 22:28:25.314133 21174 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:28:25 dist-test-slave-gjrz krb5kdc[19896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487305, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:28:25.318430 21174 init.cc:377] Logged in from keytab as kudu/127.29.106.2@KRBTEST.COM (short username kudu)
I20260811 22:28:25.335880 21174 webserver.cc:460] Webserver started at http://127.29.106.2:43697/ using document root <none> and password file <none>
I20260811 22:28:25.337070 21174 fs_manager.cc:359] Metadata directory not provided
I20260811 22:28:25.337308 21174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:28:25.337772 21174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 22:28:25 dist-test-slave-gjrz krb5kdc[19896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487305, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.2@KRBTEST.COM for HTTP/127.29.106.20@KRBTEST.COM
I20260811 22:28:25.367913 21174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-1/data/instance:
uuid: "3f852b4ade0e48c5a5b64aa38ed184cb"
format_stamp: "Formatted at 2026-08-11 22:28:25 on dist-test-slave-gjrz"
server_key: "8e51c6dce47bf524dfc5a0338720e942"
server_key_iv: "a0b9108ce4dc7468478607ebdb2ac4c0"
server_key_version: "kuduclusterkey@0"
I20260811 22:28:25.369621 21174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance:
uuid: "3f852b4ade0e48c5a5b64aa38ed184cb"
format_stamp: "Formatted at 2026-08-11 22:28:25 on dist-test-slave-gjrz"
server_key: "8e51c6dce47bf524dfc5a0338720e942"
server_key_iv: "a0b9108ce4dc7468478607ebdb2ac4c0"
server_key_version: "kuduclusterkey@0"
I20260811 22:28:25.375134 21174 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260811 22:28:25.399593 21227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:28:25.401150 21174 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 22:28:25.401314 21174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-1/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-1/wal
uuid: "3f852b4ade0e48c5a5b64aa38ed184cb"
format_stamp: "Formatted at 2026-08-11 22:28:25 on dist-test-slave-gjrz"
server_key: "8e51c6dce47bf524dfc5a0338720e942"
server_key_iv: "a0b9108ce4dc7468478607ebdb2ac4c0"
server_key_version: "kuduclusterkey@0"
I20260811 22:28:25.401547 21174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:28:25.435238 21174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:28:25.438794 21174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:28:25.439088 21174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:28:25.440057 21174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:28:25.441376 21174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:28:25.441454 21174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:28:25.441581 21174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:28:25.441663 21174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:28:25.470835 21174 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.2:39987
I20260811 22:28:25.470875 21437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.2:39987 every 8 connection(s)
I20260811 22:28:25.472066 21174 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
I20260811 22:28:25.473635 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 21174
I20260811 22:28:25.473795 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance
I20260811 22:28:25.474334 30120 external_mini_cluster.cc:1428] Setting key a47becf6ce51df0ef5ef8a19ad0ac368
Aug 11 22:28:25 dist-test-slave-gjrz krb5kdc[19896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487305, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.2@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:28:25.492743 21445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:25.474453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:56455 (local address 127.29.106.62:36269)
0811 22:28:25.474943 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:25.474947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:25.476187 (+ 1240us) server_negotiation.cc:407] Connection header received
0811 22:28:25.477858 (+ 1671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:25.477864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:25.477960 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:25.478052 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:28:25.479455 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:25.480207 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:25.482454 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:25.482861 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:25.482956 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:25.486442 (+ 3486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:28:25.486448 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:28:25.486453 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:28:25.486503 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:28:25.489120 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:25.489929 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:25.489933 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:25.489935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:25.490053 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:25.490452 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:25.490455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:25.490457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:25.490738 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:28:25.490876 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:25.491822 (+ 946us) server_negotiation.cc:299] Negotiation successful
0811 22:28:25.492164 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":236,"threads_started":1}
I20260811 22:28:25.493264 21446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:25.475089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:36269 (local address 127.29.106.2:56455)
0811 22:28:25.475942 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:25.475987 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 22:28:25.477606 (+ 1619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:28:25.478297 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:28:25.478306 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:28:25.478710 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:28:25.479288 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:25.479315 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:25.480399 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:25.480409 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:25.482279 (+ 1870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:25.482305 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:25.483072 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:25.483080 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:25.483394 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:25.484016 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:28:25.484044 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:28:25.486231 (+ 2187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:28:25.489260 (+ 3029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:28:25.489269 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:28:25.489283 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:28:25.489830 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:28:25.490145 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:28:25.490149 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:28:25.490152 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:28:25.490363 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:28:25.490920 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:28:25.490927 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:28:25.491299 (+ 372us) client_negotiation.cc:769] Sending connection context
0811 22:28:25.491577 (+ 278us) client_negotiation.cc:240] Negotiation successful
0811 22:28:25.492058 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":356,"threads_started":1}
I20260811 22:28:25.494805 21439 heartbeater.cc:347] Connected to a master server at 127.29.106.62:36269
I20260811 22:28:25.495316 21439 heartbeater.cc:464] Registering TS with master...
I20260811 22:28:25.496143 21439 heartbeater.cc:511] Master 127.29.106.62:36269 requested a full tablet report, sending...
I20260811 22:28:25.498175 20757 ts_manager.cc:194] Registered new tserver with Master: 3f852b4ade0e48c5a5b64aa38ed184cb (127.29.106.2:39987)
I20260811 22:28:25.499315 20757 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.2@KRBTEST.COM'} at 127.29.106.2:56455
WARNING: no policy specified for kudu/127.29.106.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab.
Entry for principal kudu/127.29.106.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab.
I20260811 22:28:25.534350 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.106.20:58037/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.3:0
--local_ip_for_outbound_sockets=127.29.106.3
--webserver_interface=127.29.106.3
--webserver_port=0
--tserver_master_addrs=127.29.106.62:36269
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab
--principal=kudu/127.29.106.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:41611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 22:28:25.695957 21164 heartbeater.cc:503] Master 127.29.106.62:36269 was elected leader, sending a full tablet report...
W20260811 22:28:25.858806 21452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:28:25.859071 21452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:28:25.859197 21452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:28:25.866232 21452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:28:25.866345 21452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:28:25.866482 21452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.3
I20260811 22:28:25.876354 21452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:41611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.29.106.20:58037/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab
--principal=kudu/127.29.106.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.106.3
--webserver_port=0
--tserver_master_addrs=127.29.106.62:36269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.3
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:28:25.877594 21452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:28:25.878834 21452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:28:25.891739 21463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:28:25.891794 21467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:28:25.891839 21464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:28:25.892781 21452 server_base.cc:1034] running on GCE node
I20260811 22:28:25.893368 21452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:28:25.894268 21452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:28:25.895457 21452 hybrid_clock.cc:648] HybridClock initialized: now 1786487305895407 us; error 40 us; skew 500 ppm
I20260811 22:28:25.895792 21452 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:28:25 dist-test-slave-gjrz krb5kdc[19896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487305, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:28:25.904426 21452 init.cc:377] Logged in from keytab as kudu/127.29.106.3@KRBTEST.COM (short username kudu)
I20260811 22:28:25.905907 21452 webserver.cc:460] Webserver started at http://127.29.106.3:36463/ using document root <none> and password file <none>
I20260811 22:28:25.906600 21452 fs_manager.cc:359] Metadata directory not provided
I20260811 22:28:25.906726 21452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:28:25.907038 21452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 22:28:25 dist-test-slave-gjrz krb5kdc[19896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487305, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.3@KRBTEST.COM for HTTP/127.29.106.20@KRBTEST.COM
I20260811 22:28:25.920903 21452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-2/data/instance:
uuid: "5401f173413b42f8a13e745dcfa51324"
format_stamp: "Formatted at 2026-08-11 22:28:25 on dist-test-slave-gjrz"
server_key: "a40adf25175c2ddbf9ecc6c11a647818"
server_key_iv: "b1f3db6f7846104758c760e2a4f83c5a"
server_key_version: "kuduclusterkey@0"
I20260811 22:28:25.921725 21452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance:
uuid: "5401f173413b42f8a13e745dcfa51324"
format_stamp: "Formatted at 2026-08-11 22:28:25 on dist-test-slave-gjrz"
server_key: "a40adf25175c2ddbf9ecc6c11a647818"
server_key_iv: "b1f3db6f7846104758c760e2a4f83c5a"
server_key_version: "kuduclusterkey@0"
I20260811 22:28:25.927084 21452 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260811 22:28:25.941354 21477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:28:25.942842 21452 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 22:28:25.943040 21452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-2/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-2/wal
uuid: "5401f173413b42f8a13e745dcfa51324"
format_stamp: "Formatted at 2026-08-11 22:28:25 on dist-test-slave-gjrz"
server_key: "a40adf25175c2ddbf9ecc6c11a647818"
server_key_iv: "b1f3db6f7846104758c760e2a4f83c5a"
server_key_version: "kuduclusterkey@0"
I20260811 22:28:25.943236 21452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:28:25.954924 21452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:28:25.958356 21452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:28:25.958617 21452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:28:25.959640 21452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:28:25.961020 21452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:28:25.961098 21452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:28:25.961215 21452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:28:25.961269 21452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:28:25.993324 21452 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.3:34507
I20260811 22:28:25.993366 21696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.3:34507 every 8 connection(s)
I20260811 22:28:25.994585 21452 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
I20260811 22:28:25.998700 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 21452
I20260811 22:28:25.998879 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance
I20260811 22:28:25.999519 30120 external_mini_cluster.cc:1428] Setting key 8e20f50f3d7607f1d3c6eceb304e5232
Aug 11 22:28:26 dist-test-slave-gjrz krb5kdc[19896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487305, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.3@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:28:26.013970 21703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:25.997078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:42199 (local address 127.29.106.62:36269)
0811 22:28:25.997505 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:25.997510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:25.998400 (+ 890us) server_negotiation.cc:407] Connection header received
0811 22:28:26.000809 (+ 2409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:26.000814 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:26.000918 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:26.001047 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:28:26.002677 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:26.003427 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:26.005017 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:26.005466 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:26.005562 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:26.008692 (+ 3130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:28:26.008697 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:28:26.008701 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:28:26.008740 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:28:26.011119 (+ 2379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:26.011780 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:26.011784 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:26.011786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:26.011907 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:26.012258 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:26.012260 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:26.012262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:26.012502 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:28:26.012626 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:26.013388 (+ 762us) server_negotiation.cc:299] Negotiation successful
0811 22:28:26.013599 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":219,"threads_started":1}
I20260811 22:28:26.014897 21705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:25.997523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:36269 (local address 127.29.106.3:42199)
0811 22:28:25.998222 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:25.998264 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0811 22:28:26.000548 (+ 2284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:28:26.001308 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:28:26.001319 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:28:26.001865 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:28:26.002502 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:26.002529 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:26.003565 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:26.003569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:26.004838 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:26.004850 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:26.005637 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:26.005641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:26.005835 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:26.006461 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:28:26.006483 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:28:26.008493 (+ 2010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:28:26.011240 (+ 2747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:28:26.011247 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:28:26.011259 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:28:26.011695 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:28:26.011984 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:28:26.011987 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:28:26.011989 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:28:26.012175 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:28:26.012602 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:28:26.012607 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:28:26.012894 (+ 287us) client_negotiation.cc:769] Sending connection context
0811 22:28:26.013225 (+ 331us) client_negotiation.cc:240] Negotiation successful
0811 22:28:26.013674 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":287,"threads_started":1}
I20260811 22:28:26.016096 21701 heartbeater.cc:347] Connected to a master server at 127.29.106.62:36269
I20260811 22:28:26.016363 21701 heartbeater.cc:464] Registering TS with master...
I20260811 22:28:26.017127 21701 heartbeater.cc:511] Master 127.29.106.62:36269 requested a full tablet report, sending...
I20260811 22:28:26.018530 20757 ts_manager.cc:194] Registered new tserver with Master: 5401f173413b42f8a13e745dcfa51324 (127.29.106.3:34507)
I20260811 22:28:26.019351 20757 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.3@KRBTEST.COM'} at 127.29.106.3:42199
I20260811 22:28:26.027096 30120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 22:28:26.028817 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 20912
Aug 11 22:28:26 dist-test-slave-gjrz krb5kdc[19896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.1@KRBTEST.COM
I20260811 22:28:26.044700 21708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:26.029638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60676 (local address 127.29.106.1:36207)
0811 22:28:26.030222 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:26.030228 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:26.030252 (+ 24us) server_negotiation.cc:407] Connection header received
0811 22:28:26.030304 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:26.030311 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:26.030474 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:26.030595 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:28:26.032126 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:26.033124 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:26.034508 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:26.035032 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:26.035140 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:26.039038 (+ 3898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:28:26.039057 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:28:26.039071 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:28:26.039106 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:28:26.042006 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:26.042697 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:26.042704 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:26.042706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:26.042860 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:26.043233 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:26.043236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:26.043238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:26.043604 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:28:26.043834 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:26.044083 (+ 249us) server_negotiation.cc:299] Negotiation successful
0811 22:28:26.044390 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":292,"threads_started":1}
I20260811 22:28:26.046063 21157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:28:26.140030 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 20912
I20260811 22:28:26.161581 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 21174
Aug 11 22:28:26 dist-test-slave-gjrz krb5kdc[19896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.2@KRBTEST.COM
I20260811 22:28:26.176734 21710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:26.162468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46714 (local address 127.29.106.2:39987)
0811 22:28:26.162975 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:26.162983 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:26.163010 (+ 27us) server_negotiation.cc:407] Connection header received
0811 22:28:26.163075 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:26.163082 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:26.163364 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:26.163508 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:28:26.164722 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:26.165676 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:26.166908 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:26.167504 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:26.167609 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:26.171727 (+ 4118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:28:26.171739 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:28:26.171753 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:28:26.171793 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:28:26.174324 (+ 2531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:26.174946 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:26.174952 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:26.174955 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:26.175086 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:26.175406 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:26.175409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:26.175411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:26.175749 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:28:26.175986 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:26.176170 (+ 184us) server_negotiation.cc:299] Negotiation successful
0811 22:28:26.176369 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":250,"threads_started":1}
I20260811 22:28:26.177919 21433 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:28:26.258915 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 21174
I20260811 22:28:26.283252 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 21452
Aug 11 22:28:26 dist-test-slave-gjrz krb5kdc[19896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.3@KRBTEST.COM
I20260811 22:28:26.299134 21713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:26.284227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57874 (local address 127.29.106.3:34507)
0811 22:28:26.284807 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:26.284814 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:26.284844 (+ 30us) server_negotiation.cc:407] Connection header received
0811 22:28:26.284906 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:26.284912 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:26.285129 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:26.285265 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:28:26.286485 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:26.287694 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:26.289015 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:26.289655 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:26.289761 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:26.293723 (+ 3962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:28:26.293734 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:28:26.293748 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:28:26.293781 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:28:26.296495 (+ 2714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:26.297155 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:26.297162 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:26.297165 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:26.297295 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:26.297637 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:26.297640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:26.297642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:26.297951 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:28:26.298253 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:26.298550 (+ 297us) server_negotiation.cc:299] Negotiation successful
0811 22:28:26.298792 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":275,"threads_started":1}
I20260811 22:28:26.300426 21688 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:28:26.381570 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 21452
I20260811 22:28:26.406575 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 20700
I20260811 22:28:26.407823 20819 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:28:26.513001 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 20700
I20260811 22:28:26.547755 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:36269
--webserver_interface=127.29.106.62
--webserver_port=40715
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:41611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:36269
--ranger_config_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.106.20:58037/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.29.106.20:58037/kms with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:28:26.869046 21716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:28:26.869303 21716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:28:26.869395 21716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:28:26.876230 21716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:28:26.876323 21716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:28:26.876375 21716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:28:26.876428 21716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:28:26.876475 21716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:28:26.886015 21716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:41611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.29.106.20:58037/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:36269
--ranger_config_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:36269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.106.62
--webserver_port=40715
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:28:26.887064 21716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:28:26.888264 21716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:28:26.899241 21729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:28:26.899117 21733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:28:26.899232 21731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:28:26.900307 21716 server_base.cc:1034] running on GCE node
I20260811 22:28:26.900985 21716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:28:26.902253 21716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:28:26.903438 21716 hybrid_clock.cc:648] HybridClock initialized: now 1786487306903399 us; error 26 us; skew 500 ppm
I20260811 22:28:26.903755 21716 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:28:26 dist-test-slave-gjrz krb5kdc[19896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487306, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:28:26.906652 21716 init.cc:377] Logged in from keytab as kudu/127.29.106.62@KRBTEST.COM (short username kudu)
I20260811 22:28:26.908056 21716 webserver.cc:460] Webserver started at http://127.29.106.62:40715/ using document root <none> and password file <none>
I20260811 22:28:26.908617 21716 fs_manager.cc:359] Metadata directory not provided
I20260811 22:28:26.908720 21716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 11 22:28:26 dist-test-slave-gjrz krb5kdc[19896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487306, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for HTTP/127.29.106.20@KRBTEST.COM
I20260811 22:28:26.984838 21716 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260811 22:28:26.988382 21752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:28:26.989719 21716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260811 22:28:26.989885 21716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "dbc85509642145079cb9f19ed04547ff"
format_stamp: "Formatted at 2026-08-11 22:28:23 on dist-test-slave-gjrz"
server_key: "94d3288abd16f9ded9b874bdf2232501"
server_key_iv: "04e6035cabdc0c07125b012057449441"
server_key_version: "kuduclusterkey@0"
I20260811 22:28:26.990895 21716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:28:27.025748 21716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:28:27.029609 21716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:28:27.029858 21716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:28:27.046425 21716 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.62:36269
I20260811 22:28:27.046445 21848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.62:36269 every 8 connection(s)
I20260811 22:28:27.047941 21716 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:28:27.051630 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 21716
I20260811 22:28:27.052457 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.106.20:58037/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.1:36207
--local_ip_for_outbound_sockets=127.29.106.1
--tserver_master_addrs=127.29.106.62:36269
--webserver_port=44053
--webserver_interface=127.29.106.1
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab
--principal=kudu/127.29.106.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:41611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.29.106.20:58037/kms with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 22:28:27.054674 21850 sys_catalog.cc:263] Verifying existing consensus state
I20260811 22:28:27.057816 21850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbc85509642145079cb9f19ed04547ff: Bootstrap starting.
I20260811 22:28:27.075717 21850 log.cc:824] T 00000000000000000000000000000000 P dbc85509642145079cb9f19ed04547ff: Log is configured to *not* fsync() on all Append() calls
I20260811 22:28:27.084373 21850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbc85509642145079cb9f19ed04547ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 22:28:27.084849 21850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbc85509642145079cb9f19ed04547ff: Bootstrap complete.
I20260811 22:28:27.093433 21850 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbc85509642145079cb9f19ed04547ff [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbc85509642145079cb9f19ed04547ff" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 36269 } }
I20260811 22:28:27.094367 21850 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbc85509642145079cb9f19ed04547ff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbc85509642145079cb9f19ed04547ff, State: Initialized, Role: FOLLOWER
I20260811 22:28:27.095034 21850 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbc85509642145079cb9f19ed04547ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbc85509642145079cb9f19ed04547ff" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 36269 } }
I20260811 22:28:27.095292 21850 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbc85509642145079cb9f19ed04547ff [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:28:27.095415 21850 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbc85509642145079cb9f19ed04547ff [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:28:27.095579 21850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbc85509642145079cb9f19ed04547ff [term 1 FOLLOWER]: Advancing to term 2
I20260811 22:28:27.099129 21850 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbc85509642145079cb9f19ed04547ff [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbc85509642145079cb9f19ed04547ff" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 36269 } }
I20260811 22:28:27.099650 21850 leader_election.cc:302] T 00000000000000000000000000000000 P dbc85509642145079cb9f19ed04547ff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbc85509642145079cb9f19ed04547ff; no voters:
I20260811 22:28:27.100232 21850 leader_election.cc:288] T 00000000000000000000000000000000 P dbc85509642145079cb9f19ed04547ff [CANDIDATE]: Term 2 election: Requested vote from peers
I20260811 22:28:27.100356 21858 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbc85509642145079cb9f19ed04547ff [term 2 FOLLOWER]: Leader election won for term 2
I20260811 22:28:27.102226 21850 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbc85509642145079cb9f19ed04547ff [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:28:27.105587 21858 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbc85509642145079cb9f19ed04547ff [term 2 LEADER]: Becoming Leader. State: Replica: dbc85509642145079cb9f19ed04547ff, State: Running, Role: LEADER
I20260811 22:28:27.106195 21858 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbc85509642145079cb9f19ed04547ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbc85509642145079cb9f19ed04547ff" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 36269 } }
I20260811 22:28:27.108994 21862 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbc85509642145079cb9f19ed04547ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbc85509642145079cb9f19ed04547ff. Latest consensus state: current_term: 2 leader_uuid: "dbc85509642145079cb9f19ed04547ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbc85509642145079cb9f19ed04547ff" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 36269 } } }
I20260811 22:28:27.109380 21862 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbc85509642145079cb9f19ed04547ff [sys.catalog]: This master's current role is: LEADER
I20260811 22:28:27.109812 21861 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbc85509642145079cb9f19ed04547ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "dbc85509642145079cb9f19ed04547ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbc85509642145079cb9f19ed04547ff" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 36269 } } }
I20260811 22:28:27.110076 21861 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbc85509642145079cb9f19ed04547ff [sys.catalog]: This master's current role is: LEADER
I20260811 22:28:27.116490 21882 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:28:27.122289 21882 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:28:27.124835 21882 catalog_manager.cc:1259] Loaded cluster ID: 0cccbdbdee424760a73815c776a42016
I20260811 22:28:27.124933 21882 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:28:27.128953 21882 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:28:27.130867 21882 catalog_manager.cc:5879] T 00000000000000000000000000000000 P dbc85509642145079cb9f19ed04547ff: Loaded TSK: 0
I20260811 22:28:27.132405 21882 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:28:27.134788 21850 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260811 22:28:27.496996 21854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:28:27.497359 21854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:28:27.497568 21854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:28:27.508791 21854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:28:27.509001 21854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:28:27.509241 21854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.1
I20260811 22:28:27.524340 21854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:41611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.29.106.20:58037/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab
--principal=kudu/127.29.106.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.1:36207
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.106.1
--webserver_port=44053
--tserver_master_addrs=127.29.106.62:36269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.1
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:28:27.526269 21854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:28:27.528126 21854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:28:27.571421 21947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:28:27.575196 21938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:28:27.575690 21951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:28:27.578908 21854 server_base.cc:1034] running on GCE node
I20260811 22:28:27.579692 21854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:28:27.582940 21854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:28:27.584247 21854 hybrid_clock.cc:648] HybridClock initialized: now 1786487307584170 us; error 65 us; skew 500 ppm
I20260811 22:28:27.584703 21854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:28:27 dist-test-slave-gjrz krb5kdc[19896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487307, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:28:27.588930 21854 init.cc:377] Logged in from keytab as kudu/127.29.106.1@KRBTEST.COM (short username kudu)
I20260811 22:28:27.611116 21854 webserver.cc:460] Webserver started at http://127.29.106.1:44053/ using document root <none> and password file <none>
I20260811 22:28:27.612149 21854 fs_manager.cc:359] Metadata directory not provided
I20260811 22:28:27.612370 21854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 11 22:28:27 dist-test-slave-gjrz krb5kdc[19896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487307, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.1@KRBTEST.COM for HTTP/127.29.106.20@KRBTEST.COM
I20260811 22:28:27.665093 21854 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.002s
I20260811 22:28:27.677253 21974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:28:27.678727 21854 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.007s sys 0.006s
I20260811 22:28:27.678908 21854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-0/wal
uuid: "05a49b25fe69430aa8303b2c5659158d"
format_stamp: "Formatted at 2026-08-11 22:28:24 on dist-test-slave-gjrz"
server_key: "d13343c28432478b5fdac0bfcb4d42ae"
server_key_iv: "4bea3e1e50c4447fef23a5cce0517b34"
server_key_version: "kuduclusterkey@0"
I20260811 22:28:27.679966 21854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:28:27.729175 21854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:28:27.733687 21854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:28:27.733963 21854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:28:27.734973 21854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:28:27.736364 21854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:28:27.736465 21854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:28:27.736577 21854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:28:27.736675 21854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:28:27.791016 21854 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.1:36207
I20260811 22:28:27.791101 22200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.1:36207 every 8 connection(s)
I20260811 22:28:27.792881 21854 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
I20260811 22:28:27.793886 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 21854
I20260811 22:28:27.794730 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.106.20:58037/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.2:39987
--local_ip_for_outbound_sockets=127.29.106.2
--tserver_master_addrs=127.29.106.62:36269
--webserver_port=43697
--webserver_interface=127.29.106.2
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab
--principal=kudu/127.29.106.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:41611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.29.106.20:58037/kms with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 22:28:27 dist-test-slave-gjrz krb5kdc[19896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487307, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.1@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:28:27.843230 22204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:27.797882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:36269 (local address 127.29.106.1:45145)
0811 22:28:27.815499 (+ 17617us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:27.815544 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 22:28:27.817941 (+ 2397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:28:27.820270 (+ 2329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:28:27.820280 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:28:27.820908 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:28:27.821653 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:27.821677 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:27.824135 (+ 2458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:27.824140 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:27.826089 (+ 1949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:27.826104 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:27.827834 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:27.827841 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:27.828121 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:27.829985 (+ 1864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:28:27.830014 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:28:27.833216 (+ 3202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:28:27.838492 (+ 5276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:28:27.838501 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:28:27.838525 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:28:27.839119 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:28:27.839730 (+ 611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:28:27.839735 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:28:27.839739 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:28:27.840000 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:28:27.840792 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:28:27.840801 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:28:27.841116 (+ 315us) client_negotiation.cc:769] Sending connection context
0811 22:28:27.841349 (+ 233us) client_negotiation.cc:240] Negotiation successful
0811 22:28:27.841917 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17319,"thread_start_us":17092,"threads_started":1}
I20260811 22:28:27.843227 22205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:27.798952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:45145 (local address 127.29.106.62:36269)
0811 22:28:27.814872 (+ 15920us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:27.814894 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:27.815962 (+ 1068us) server_negotiation.cc:407] Connection header received
0811 22:28:27.818626 (+ 2664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:27.818645 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:27.819100 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:27.819486 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:28:27.821908 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:27.823915 (+ 2007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:27.826248 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:27.827294 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:27.827439 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:27.833511 (+ 6072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:28:27.833522 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:28:27.833541 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:28:27.833595 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:28:27.838292 (+ 4697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:27.839323 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:27.839332 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:27.839340 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:27.839557 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:27.840161 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:27.840167 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:27.840171 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:27.840639 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:28:27.840833 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:27.841617 (+ 784us) server_negotiation.cc:299] Negotiation successful
0811 22:28:27.842034 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15643,"thread_start_us":15418,"threads_started":1}
I20260811 22:28:27.846575 22201 heartbeater.cc:347] Connected to a master server at 127.29.106.62:36269
I20260811 22:28:27.846879 22201 heartbeater.cc:464] Registering TS with master...
I20260811 22:28:27.847788 22201 heartbeater.cc:511] Master 127.29.106.62:36269 requested a full tablet report, sending...
W20260811 22:28:27.851056 22201 heartbeater.cc:644] Failed to heartbeat to 127.29.106.62:36269 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260811 22:28:27.851423 22201 heartbeater.cc:464] Registering TS with master...
I20260811 22:28:27.851692 22201 heartbeater.cc:511] Master 127.29.106.62:36269 requested a full tablet report, sending...
I20260811 22:28:27.852897 22201 heartbeater.cc:464] Registering TS with master...
I20260811 22:28:27.853147 22201 heartbeater.cc:511] Master 127.29.106.62:36269 requested a full tablet report, sending...
W20260811 22:28:27.854220 22201 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260811 22:28:28.211179 22208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:28:28.211620 22208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:28:28.211787 22208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:28:28.222806 22208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:28:28.222998 22208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:28:28.223217 22208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.2
I20260811 22:28:28.237758 22208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:41611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.29.106.20:58037/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab
--principal=kudu/127.29.106.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.2:39987
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.106.2
--webserver_port=43697
--tserver_master_addrs=127.29.106.62:36269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.2
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:28:28.239033 22208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:28:28.240371 22208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:28:28.257386 22234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:28:28.258339 22235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:28:28.272830 22240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:28:28.273185 22208 server_base.cc:1034] running on GCE node
I20260811 22:28:28.273871 22208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:28:28.274888 22208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:28:28.276190 22208 hybrid_clock.cc:648] HybridClock initialized: now 1786487308276130 us; error 50 us; skew 500 ppm
I20260811 22:28:28.276510 22208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:28:28 dist-test-slave-gjrz krb5kdc[19896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487308, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:28:28.286556 22208 init.cc:377] Logged in from keytab as kudu/127.29.106.2@KRBTEST.COM (short username kudu)
I20260811 22:28:28.289517 22208 webserver.cc:460] Webserver started at http://127.29.106.2:43697/ using document root <none> and password file <none>
I20260811 22:28:28.290239 22208 fs_manager.cc:359] Metadata directory not provided
I20260811 22:28:28.290372 22208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 11 22:28:28 dist-test-slave-gjrz krb5kdc[19896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487308, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.2@KRBTEST.COM for HTTP/127.29.106.20@KRBTEST.COM
Aug 11 22:28:28 dist-test-slave-gjrz krb5kdc[19896](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786487308, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:28:28.318984 22208 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.003s sys 0.000s
I20260811 22:28:28.321851 22248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:28:28.323700 22208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260811 22:28:28.323905 22208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-1/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-1/wal
uuid: "3f852b4ade0e48c5a5b64aa38ed184cb"
format_stamp: "Formatted at 2026-08-11 22:28:25 on dist-test-slave-gjrz"
server_key: "8e51c6dce47bf524dfc5a0338720e942"
server_key_iv: "a0b9108ce4dc7468478607ebdb2ac4c0"
server_key_version: "kuduclusterkey@0"
I20260811 22:28:28.325389 22208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:28:28.359050 22208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:28:28.363708 22208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:28:28.363983 22208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:28:28.365185 22208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:28:28.367004 22208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:28:28.367125 22208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:28:28.367242 22208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:28:28.367357 22208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260811 22:28:28.409739 22208 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.2:39987
I20260811 22:28:28.409821 22476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.2:39987 every 8 connection(s)
I20260811 22:28:28.411692 22208 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
I20260811 22:28:28.421324 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 22208
I20260811 22:28:28.421949 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.106.20:58037/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.3:34507
--local_ip_for_outbound_sockets=127.29.106.3
--tserver_master_addrs=127.29.106.62:36269
--webserver_port=36463
--webserver_interface=127.29.106.3
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab
--principal=kudu/127.29.106.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:41611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.29.106.20:58037/kms with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 22:28:28 dist-test-slave-gjrz krb5kdc[19896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487308, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.2@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:28:28.459802 22481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:28.416678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:57171 (local address 127.29.106.62:36269)
0811 22:28:28.440551 (+ 23873us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:28.440558 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:28.440577 (+ 19us) server_negotiation.cc:407] Connection header received
0811 22:28:28.440637 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:28.440641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:28.440711 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:28.440807 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:28:28.443533 (+ 2726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:28.444582 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:28.446810 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:28.447387 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:28.447490 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:28.452133 (+ 4643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:28:28.452140 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:28:28.452145 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:28:28.452187 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:28:28.455946 (+ 3759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:28.456933 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:28.456939 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:28.456942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:28.457117 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:28.457705 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:28.457711 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:28.457714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:28.458084 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:28:28.458214 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:28.459160 (+ 946us) server_negotiation.cc:299] Negotiation successful
0811 22:28:28.459415 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23757,"thread_start_us":23654,"threads_started":1}
I20260811 22:28:28.462255 22480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:28.415110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:36269 (local address 127.29.106.2:57171)
0811 22:28:28.437260 (+ 22150us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:28.437319 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0811 22:28:28.439865 (+ 2546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:28:28.441250 (+ 1385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:28:28.441261 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:28:28.441920 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:28:28.442749 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:28.442773 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:28.444777 (+ 2004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:28.444781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:28.446596 (+ 1815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:28.446628 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:28.447786 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:28.447796 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:28.448106 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:28.449010 (+ 904us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:28:28.449039 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:28:28.451796 (+ 2757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:28:28.456134 (+ 4338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:28:28.456143 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:28:28.456160 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:28:28.456778 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:28:28.457276 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:28:28.457281 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:28:28.457284 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:28:28.457560 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:28:28.458304 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:28:28.458312 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:28:28.458715 (+ 403us) client_negotiation.cc:769] Sending connection context
0811 22:28:28.460343 (+ 1628us) client_negotiation.cc:240] Negotiation successful
0811 22:28:28.460894 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21781,"thread_start_us":24030,"threads_started":1}
I20260811 22:28:28.464028 22478 heartbeater.cc:347] Connected to a master server at 127.29.106.62:36269
I20260811 22:28:28.464327 22478 heartbeater.cc:464] Registering TS with master...
I20260811 22:28:28.465198 22478 heartbeater.cc:511] Master 127.29.106.62:36269 requested a full tablet report, sending...
I20260811 22:28:28.467232 21779 ts_manager.cc:194] Registered new tserver with Master: 3f852b4ade0e48c5a5b64aa38ed184cb (127.29.106.2:39987)
I20260811 22:28:28.468827 21779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.2@KRBTEST.COM'} at 127.29.106.2:57171
Aug 11 22:28:28 dist-test-slave-gjrz krb5kdc[19896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487308, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for HTTP/127.29.106.20@KRBTEST.COM
I20260811 22:28:28.854385 22201 heartbeater.cc:464] Registering TS with master...
I20260811 22:28:28.854832 22201 heartbeater.cc:511] Master 127.29.106.62:36269 requested a full tablet report, sending...
I20260811 22:28:28.856281 21779 ts_manager.cc:194] Registered new tserver with Master: 05a49b25fe69430aa8303b2c5659158d (127.29.106.1:36207)
I20260811 22:28:28.857512 21779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.1@KRBTEST.COM'} at 127.29.106.1:45145
W20260811 22:28:28.879956 22484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:28:28.880291 22484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:28:28.880487 22484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:28:28.891589 22484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:28:28.891728 22484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:28:28.891963 22484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.3
I20260811 22:28:28.906908 22484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:41611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.29.106.20:58037/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab
--principal=kudu/127.29.106.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.3:34507
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.106.3
--webserver_port=36463
--tserver_master_addrs=127.29.106.62:36269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.3
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:28:28.908526 22484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:28:28.910290 22484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:28:28.926354 22492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:28:28.927094 22493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:28:28.944568 22498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:28:28.945648 22484 server_base.cc:1034] running on GCE node
I20260811 22:28:28.946368 22484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:28:28.947535 22484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:28:28.948827 22484 hybrid_clock.cc:648] HybridClock initialized: now 1786487308948751 us; error 60 us; skew 500 ppm
I20260811 22:28:28.949201 22484 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:28:28 dist-test-slave-gjrz krb5kdc[19896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487308, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:28:28.953258 22484 init.cc:377] Logged in from keytab as kudu/127.29.106.3@KRBTEST.COM (short username kudu)
I20260811 22:28:28.955188 22484 webserver.cc:460] Webserver started at http://127.29.106.3:36463/ using document root <none> and password file <none>
I20260811 22:28:28.956126 22484 fs_manager.cc:359] Metadata directory not provided
I20260811 22:28:28.956276 22484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 11 22:28:28 dist-test-slave-gjrz krb5kdc[19896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487308, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.3@KRBTEST.COM for HTTP/127.29.106.20@KRBTEST.COM
I20260811 22:28:28.984447 22484 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.009s sys 0.000s
I20260811 22:28:28.987833 22515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:28:28.989526 22484 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260811 22:28:28.989682 22484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-2/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-2/wal
uuid: "5401f173413b42f8a13e745dcfa51324"
format_stamp: "Formatted at 2026-08-11 22:28:25 on dist-test-slave-gjrz"
server_key: "a40adf25175c2ddbf9ecc6c11a647818"
server_key_iv: "b1f3db6f7846104758c760e2a4f83c5a"
server_key_version: "kuduclusterkey@0"
I20260811 22:28:28.990955 22484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:28:29.024124 22484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:28:29.028978 22484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:28:29.029265 22484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:28:29.030386 22484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:28:29.032027 22484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:28:29.032166 22484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260811 22:28:29.032279 22484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:28:29.032358 22484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 22:28:29 dist-test-slave-gjrz krb5kdc[19896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487308, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for HTTP/127.29.106.20@KRBTEST.COM
I20260811 22:28:29.295665 22484 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.3:34507
I20260811 22:28:29.295722 22736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.3:34507 every 8 connection(s)
I20260811 22:28:29.297735 22484 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
I20260811 22:28:29.306367 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 22484
Aug 11 22:28:29 dist-test-slave-gjrz krb5kdc[19896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487308, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.3@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:28:29.325675 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:29.302730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:36269 (local address 127.29.106.3:36003)
0811 22:28:29.303531 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:29.303578 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0811 22:28:29.305278 (+ 1700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:28:29.306003 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:28:29.306011 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:28:29.306471 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:28:29.307161 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:29.307188 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:29.309010 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:29.309017 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:29.311127 (+ 2110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:29.311154 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:29.312401 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:29.312409 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:29.312722 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:29.313577 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:28:29.313611 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:28:29.316677 (+ 3066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:28:29.321085 (+ 4408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:28:29.321091 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:28:29.321105 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:28:29.321672 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:28:29.322133 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:28:29.322139 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:28:29.322142 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:28:29.322423 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:28:29.323050 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:28:29.323058 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:28:29.323529 (+ 471us) client_negotiation.cc:769] Sending connection context
0811 22:28:29.323837 (+ 308us) client_negotiation.cc:240] Negotiation successful
0811 22:28:29.324357 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":310,"threads_started":1}
I20260811 22:28:29.327019 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:29.301984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:36003 (local address 127.29.106.62:36269)
0811 22:28:29.302418 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:29.302423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:29.303685 (+ 1262us) server_negotiation.cc:407] Connection header received
0811 22:28:29.305540 (+ 1855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:29.305546 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:29.305639 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:29.305741 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:28:29.307419 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:29.308788 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:29.311395 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:29.312134 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:29.315126 (+ 2992us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:29.316969 (+ 1843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:28:29.316976 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:28:29.316982 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:28:29.317050 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:28:29.320954 (+ 3904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:29.321811 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:29.321817 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:29.321820 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:29.321998 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:29.322553 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:29.322559 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:29.322562 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:29.322898 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:28:29.323039 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:29.326509 (+ 3470us) server_negotiation.cc:299] Negotiation successful
0811 22:28:29.326691 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":234,"threads_started":1}
I20260811 22:28:29.328248 22741 heartbeater.cc:347] Connected to a master server at 127.29.106.62:36269
I20260811 22:28:29.328595 22741 heartbeater.cc:464] Registering TS with master...
I20260811 22:28:29.329284 22741 heartbeater.cc:511] Master 127.29.106.62:36269 requested a full tablet report, sending...
I20260811 22:28:29.330706 21779 ts_manager.cc:194] Registered new tserver with Master: 5401f173413b42f8a13e745dcfa51324 (127.29.106.3:34507)
I20260811 22:28:29.331700 21779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.3@KRBTEST.COM'} at 127.29.106.3:36003
I20260811 22:28:29.335249 22748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:29.307321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54578 (local address 127.29.106.62:36269)
0811 22:28:29.320975 (+ 13654us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:29.320981 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:29.321000 (+ 19us) server_negotiation.cc:407] Connection header received
0811 22:28:29.321059 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:29.321064 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:29.321148 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:29.321267 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:28:29.323175 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:29.324460 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:29.326342 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:29.327067 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:29.327186 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:29.330077 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:28:29.330082 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:28:29.330087 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:28:29.330128 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:28:29.333102 (+ 2974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:29.333737 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:29.333743 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:29.333746 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:29.333892 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:29.334199 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:29.334203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:29.334206 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:29.334455 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:28:29.334662 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:29.334865 (+ 203us) server_negotiation.cc:299] Negotiation successful
0811 22:28:29.335008 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13537,"thread_start_us":13392,"threads_started":1}
I20260811 22:28:29.337620 30120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 22:28:29.351855 22748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:29.340499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54582 (local address 127.29.106.62:36269)
0811 22:28:29.340658 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:29.340662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:29.341065 (+ 403us) server_negotiation.cc:407] Connection header received
0811 22:28:29.341167 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:29.341171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:29.341222 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:29.341292 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:28:29.342714 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:29.343402 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:29.344710 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:29.345108 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:29.345190 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:29.347444 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:28:29.347448 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:28:29.347452 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:28:29.347478 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:28:29.349791 (+ 2313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:29.350345 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:29.350348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:29.350350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:29.350457 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:29.350729 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:29.350732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:29.350734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:29.350952 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:28:29.351123 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:29.351428 (+ 305us) server_negotiation.cc:299] Negotiation successful
0811 22:28:29.351576 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260811 22:28:29.356201 21779 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54582:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:28:29.363674 21779 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:28:29.394245 22781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:29.377143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57890 (local address 127.29.106.3:34507)
0811 22:28:29.377845 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:29.377853 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:29.378563 (+ 710us) server_negotiation.cc:407] Connection header received
0811 22:28:29.378830 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:29.378837 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:29.379081 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:29.379243 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:28:29.383604 (+ 4361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:29.385426 (+ 1822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:29.389620 (+ 4194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:29.391295 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:29.391406 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:29.391671 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:29.393571 (+ 1900us) server_negotiation.cc:299] Negotiation successful
0811 22:28:29.393831 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":372,"threads_started":1}
I20260811 22:28:29.395324 22785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:29.376684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.3:34507 (local address 127.0.0.1:57890)
0811 22:28:29.378415 (+ 1731us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:29.378444 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0811 22:28:29.378634 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:28:29.382793 (+ 4159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:28:29.382802 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:28:29.382839 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:28:29.383353 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:29.383367 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:29.386738 (+ 3371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:29.386744 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:29.389382 (+ 2638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:29.389407 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:29.393030 (+ 3623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:29.393036 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:29.393272 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:29.393326 (+ 54us) client_negotiation.cc:769] Sending connection context
0811 22:28:29.394598 (+ 1272us) client_negotiation.cc:240] Negotiation successful
0811 22:28:29.394883 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1527,"thread_start_us":1367,"threads_started":1}
I20260811 22:28:29.399369 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:29.377861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:36207 (local address 127.0.0.1:60692)
0811 22:28:29.389500 (+ 11639us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:29.389521 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 22:28:29.389645 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:28:29.391199 (+ 1554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:28:29.391204 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:28:29.391234 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:28:29.391636 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:29.391648 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:29.393774 (+ 2126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:29.393780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:29.397150 (+ 3370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:29.397162 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:29.398693 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:29.398699 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:29.398859 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:29.398893 (+ 34us) client_negotiation.cc:769] Sending connection context
0811 22:28:29.398991 (+ 98us) client_negotiation.cc:240] Negotiation successful
0811 22:28:29.399089 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11506,"thread_start_us":18749,"threads_started":1}
I20260811 22:28:29.400131 22789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:29.377312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60692 (local address 127.29.106.1:36207)
0811 22:28:29.390154 (+ 12842us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:29.390164 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:29.390198 (+ 34us) server_negotiation.cc:407] Connection header received
0811 22:28:29.390308 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:29.390318 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:29.390568 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:29.390771 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:28:29.391879 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:29.393581 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:29.397337 (+ 3756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:29.398476 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:29.398604 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:29.398824 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:29.399631 (+ 807us) server_negotiation.cc:299] Negotiation successful
0811 22:28:29.399795 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12669,"thread_start_us":12533,"threads_started":1}
I20260811 22:28:29.399771 22614 tablet_service.cc:1467] Processing CreateTablet for tablet 3d8b9e0f596f4c99b9b8fea3a84fdeac (DEFAULT_TABLE table=test-table [id=1ccb48331e4e4370ab29848e7b1c9210]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:28:29.401333 22614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d8b9e0f596f4c99b9b8fea3a84fdeac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:28:29.403321 22785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:29.378832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.2:39987 (local address 127.0.0.1:46726)
0811 22:28:29.395698 (+ 16866us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:29.395719 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 22:28:29.395823 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:28:29.397378 (+ 1555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:28:29.397383 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:28:29.397408 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:28:29.397602 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:29.397615 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:29.399404 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:29.399408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:29.400837 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:29.400850 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:29.402615 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:29.402623 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:29.402801 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:29.402832 (+ 31us) client_negotiation.cc:769] Sending connection context
0811 22:28:29.402914 (+ 82us) client_negotiation.cc:240] Negotiation successful
0811 22:28:29.403042 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16727,"thread_start_us":12144,"threads_started":1}
I20260811 22:28:29.404314 22793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:29.379089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46726 (local address 127.29.106.2:39987)
0811 22:28:29.396480 (+ 17391us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:29.396489 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:29.396516 (+ 27us) server_negotiation.cc:407] Connection header received
0811 22:28:29.396630 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:29.396638 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:29.396931 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:29.397127 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:28:29.397777 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:29.399253 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:29.400978 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:29.402387 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:29.402511 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:29.403774 (+ 1263us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:29.403851 (+ 77us) server_negotiation.cc:299] Negotiation successful
0811 22:28:29.404010 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17236,"thread_start_us":17114,"threads_started":1}
I20260811 22:28:29.403860 22073 tablet_service.cc:1467] Processing CreateTablet for tablet 3d8b9e0f596f4c99b9b8fea3a84fdeac (DEFAULT_TABLE table=test-table [id=1ccb48331e4e4370ab29848e7b1c9210]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:28:29.405115 22073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d8b9e0f596f4c99b9b8fea3a84fdeac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:28:29.408481 22346 tablet_service.cc:1467] Processing CreateTablet for tablet 3d8b9e0f596f4c99b9b8fea3a84fdeac (DEFAULT_TABLE table=test-table [id=1ccb48331e4e4370ab29848e7b1c9210]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:28:29.409876 22346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d8b9e0f596f4c99b9b8fea3a84fdeac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:28:29.419935 22795 tablet_bootstrap.cc:492] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 05a49b25fe69430aa8303b2c5659158d: Bootstrap starting.
I20260811 22:28:29.422932 22795 tablet_bootstrap.cc:654] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 05a49b25fe69430aa8303b2c5659158d: Neither blocks nor log segments found. Creating new log.
I20260811 22:28:29.424269 22795 log.cc:824] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 05a49b25fe69430aa8303b2c5659158d: Log is configured to *not* fsync() on all Append() calls
I20260811 22:28:29.424898 22794 tablet_bootstrap.cc:492] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 5401f173413b42f8a13e745dcfa51324: Bootstrap starting.
I20260811 22:28:29.426406 22797 tablet_bootstrap.cc:492] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 3f852b4ade0e48c5a5b64aa38ed184cb: Bootstrap starting.
I20260811 22:28:29.427533 22795 tablet_bootstrap.cc:492] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 05a49b25fe69430aa8303b2c5659158d: No bootstrap required, opened a new log
I20260811 22:28:29.427829 22795 ts_tablet_manager.cc:1397] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 05a49b25fe69430aa8303b2c5659158d: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260811 22:28:29.428465 22794 tablet_bootstrap.cc:654] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 5401f173413b42f8a13e745dcfa51324: Neither blocks nor log segments found. Creating new log.
I20260811 22:28:29.429211 22797 tablet_bootstrap.cc:654] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 3f852b4ade0e48c5a5b64aa38ed184cb: Neither blocks nor log segments found. Creating new log.
I20260811 22:28:29.429725 22794 log.cc:824] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 5401f173413b42f8a13e745dcfa51324: Log is configured to *not* fsync() on all Append() calls
I20260811 22:28:29.430536 22797 log.cc:824] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 3f852b4ade0e48c5a5b64aa38ed184cb: Log is configured to *not* fsync() on all Append() calls
I20260811 22:28:29.432847 22794 tablet_bootstrap.cc:492] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 5401f173413b42f8a13e745dcfa51324: No bootstrap required, opened a new log
I20260811 22:28:29.433121 22794 ts_tablet_manager.cc:1397] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 5401f173413b42f8a13e745dcfa51324: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260811 22:28:29.437255 22797 tablet_bootstrap.cc:492] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 3f852b4ade0e48c5a5b64aa38ed184cb: No bootstrap required, opened a new log
I20260811 22:28:29.437490 22797 ts_tablet_manager.cc:1397] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 3f852b4ade0e48c5a5b64aa38ed184cb: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260811 22:28:29.437647 22795 raft_consensus.cc:348] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 05a49b25fe69430aa8303b2c5659158d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5401f173413b42f8a13e745dcfa51324" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 34507 } } peers { permanent_uuid: "05a49b25fe69430aa8303b2c5659158d" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 36207 } } peers { permanent_uuid: "3f852b4ade0e48c5a5b64aa38ed184cb" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39987 } }
I20260811 22:28:29.438074 22795 raft_consensus.cc:374] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 05a49b25fe69430aa8303b2c5659158d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:28:29.438179 22795 raft_consensus.cc:729] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 05a49b25fe69430aa8303b2c5659158d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05a49b25fe69430aa8303b2c5659158d, State: Initialized, Role: FOLLOWER
I20260811 22:28:29.438792 22795 consensus_queue.cc:260] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 05a49b25fe69430aa8303b2c5659158d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5401f173413b42f8a13e745dcfa51324" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 34507 } } peers { permanent_uuid: "05a49b25fe69430aa8303b2c5659158d" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 36207 } } peers { permanent_uuid: "3f852b4ade0e48c5a5b64aa38ed184cb" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39987 } }
I20260811 22:28:29.439934 22201 heartbeater.cc:503] Master 127.29.106.62:36269 was elected leader, sending a full tablet report...
I20260811 22:28:29.440521 22795 ts_tablet_manager.cc:1428] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 05a49b25fe69430aa8303b2c5659158d: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260811 22:28:29.442951 22794 raft_consensus.cc:348] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 5401f173413b42f8a13e745dcfa51324 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5401f173413b42f8a13e745dcfa51324" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 34507 } } peers { permanent_uuid: "05a49b25fe69430aa8303b2c5659158d" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 36207 } } peers { permanent_uuid: "3f852b4ade0e48c5a5b64aa38ed184cb" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39987 } }
I20260811 22:28:29.443298 22794 raft_consensus.cc:374] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 5401f173413b42f8a13e745dcfa51324 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:28:29.443413 22794 raft_consensus.cc:729] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 5401f173413b42f8a13e745dcfa51324 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5401f173413b42f8a13e745dcfa51324, State: Initialized, Role: FOLLOWER
I20260811 22:28:29.443974 22794 consensus_queue.cc:260] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 5401f173413b42f8a13e745dcfa51324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5401f173413b42f8a13e745dcfa51324" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 34507 } } peers { permanent_uuid: "05a49b25fe69430aa8303b2c5659158d" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 36207 } } peers { permanent_uuid: "3f852b4ade0e48c5a5b64aa38ed184cb" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39987 } }
I20260811 22:28:29.444957 22741 heartbeater.cc:503] Master 127.29.106.62:36269 was elected leader, sending a full tablet report...
I20260811 22:28:29.445590 22794 ts_tablet_manager.cc:1428] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 5401f173413b42f8a13e745dcfa51324: Time spent starting tablet: real 0.012s user 0.010s sys 0.003s
I20260811 22:28:29.447060 22797 raft_consensus.cc:348] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 3f852b4ade0e48c5a5b64aa38ed184cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5401f173413b42f8a13e745dcfa51324" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 34507 } } peers { permanent_uuid: "05a49b25fe69430aa8303b2c5659158d" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 36207 } } peers { permanent_uuid: "3f852b4ade0e48c5a5b64aa38ed184cb" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39987 } }
I20260811 22:28:29.447487 22797 raft_consensus.cc:374] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 3f852b4ade0e48c5a5b64aa38ed184cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:28:29.447604 22797 raft_consensus.cc:729] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 3f852b4ade0e48c5a5b64aa38ed184cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f852b4ade0e48c5a5b64aa38ed184cb, State: Initialized, Role: FOLLOWER
I20260811 22:28:29.448163 22797 consensus_queue.cc:260] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 3f852b4ade0e48c5a5b64aa38ed184cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5401f173413b42f8a13e745dcfa51324" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 34507 } } peers { permanent_uuid: "05a49b25fe69430aa8303b2c5659158d" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 36207 } } peers { permanent_uuid: "3f852b4ade0e48c5a5b64aa38ed184cb" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39987 } }
I20260811 22:28:29.448438 22803 raft_consensus.cc:482] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 05a49b25fe69430aa8303b2c5659158d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:28:29.448700 22803 raft_consensus.cc:504] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 05a49b25fe69430aa8303b2c5659158d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5401f173413b42f8a13e745dcfa51324" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 34507 } } peers { permanent_uuid: "05a49b25fe69430aa8303b2c5659158d" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 36207 } } peers { permanent_uuid: "3f852b4ade0e48c5a5b64aa38ed184cb" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39987 } }
I20260811 22:28:29.449198 22478 heartbeater.cc:503] Master 127.29.106.62:36269 was elected leader, sending a full tablet report...
I20260811 22:28:29.449819 22797 ts_tablet_manager.cc:1428] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 3f852b4ade0e48c5a5b64aa38ed184cb: Time spent starting tablet: real 0.012s user 0.012s sys 0.001s
I20260811 22:28:29.450270 22803 leader_election.cc:288] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 05a49b25fe69430aa8303b2c5659158d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5401f173413b42f8a13e745dcfa51324 (127.29.106.3:34507), 3f852b4ade0e48c5a5b64aa38ed184cb (127.29.106.2:39987)
I20260811 22:28:29.457016 22781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:29.450860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:49655 (local address 127.29.106.3:34507)
0811 22:28:29.451041 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:29.451047 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:29.451105 (+ 58us) server_negotiation.cc:407] Connection header received
0811 22:28:29.451416 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:29.451421 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:29.451490 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:29.451549 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:28:29.452392 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:29.453162 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:29.455229 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:29.456050 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:29.456194 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:29.456268 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:29.456660 (+ 392us) server_negotiation.cc:299] Negotiation successful
0811 22:28:29.456786 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260811 22:28:29.457091 22809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:29.450448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.3:34507 (local address 127.29.106.1:49655)
0811 22:28:29.450995 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:29.451015 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 22:28:29.451220 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:28:29.451746 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:28:29.451750 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:28:29.451775 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:28:29.452207 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:29.452223 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:29.453269 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:29.453275 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:29.455077 (+ 1802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:29.455090 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:29.456269 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:29.456278 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:29.456503 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:29.456534 (+ 31us) client_negotiation.cc:769] Sending connection context
0811 22:28:29.456629 (+ 95us) client_negotiation.cc:240] Negotiation successful
0811 22:28:29.456755 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":245,"threads_started":1}
I20260811 22:28:29.458011 22651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8b9e0f596f4c99b9b8fea3a84fdeac" candidate_uuid: "05a49b25fe69430aa8303b2c5659158d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5401f173413b42f8a13e745dcfa51324" is_pre_election: true
I20260811 22:28:29.458516 22651 raft_consensus.cc:2454] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 5401f173413b42f8a13e745dcfa51324 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05a49b25fe69430aa8303b2c5659158d in term 0.
I20260811 22:28:29.459196 21985 leader_election.cc:302] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 05a49b25fe69430aa8303b2c5659158d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05a49b25fe69430aa8303b2c5659158d, 5401f173413b42f8a13e745dcfa51324; no voters:
I20260811 22:28:29.459601 22803 raft_consensus.cc:2781] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 05a49b25fe69430aa8303b2c5659158d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:28:29.459746 22803 raft_consensus.cc:482] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 05a49b25fe69430aa8303b2c5659158d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:28:29.459846 22803 raft_consensus.cc:3037] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 05a49b25fe69430aa8303b2c5659158d [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:28:29.461362 22793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:29.451304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:45711 (local address 127.29.106.2:39987)
0811 22:28:29.451495 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:29.451501 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:29.455819 (+ 4318us) server_negotiation.cc:407] Connection header received
0811 22:28:29.455978 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:29.455983 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:29.456052 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:29.456126 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:28:29.456708 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:29.457556 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:29.459651 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:29.460524 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:29.460643 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:29.460695 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:29.461101 (+ 406us) server_negotiation.cc:299] Negotiation successful
0811 22:28:29.461189 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260811 22:28:29.461567 22811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:29.451103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.2:39987 (local address 127.29.106.1:45711)
0811 22:28:29.455724 (+ 4621us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:29.455738 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 22:28:29.455856 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:28:29.456155 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:28:29.456160 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:28:29.456184 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:28:29.456567 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:29.456578 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:29.457733 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:29.457738 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:29.459456 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:29.459472 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:29.460749 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:29.460756 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:29.460942 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:29.460980 (+ 38us) client_negotiation.cc:769] Sending connection context
0811 22:28:29.461091 (+ 111us) client_negotiation.cc:240] Negotiation successful
0811 22:28:29.461231 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4515,"thread_start_us":4357,"threads_started":1}
I20260811 22:28:29.462440 22377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8b9e0f596f4c99b9b8fea3a84fdeac" candidate_uuid: "05a49b25fe69430aa8303b2c5659158d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f852b4ade0e48c5a5b64aa38ed184cb" is_pre_election: true
I20260811 22:28:29.462510 22803 raft_consensus.cc:504] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 05a49b25fe69430aa8303b2c5659158d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5401f173413b42f8a13e745dcfa51324" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 34507 } } peers { permanent_uuid: "05a49b25fe69430aa8303b2c5659158d" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 36207 } } peers { permanent_uuid: "3f852b4ade0e48c5a5b64aa38ed184cb" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39987 } }
I20260811 22:28:29.462867 22377 raft_consensus.cc:2454] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 3f852b4ade0e48c5a5b64aa38ed184cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05a49b25fe69430aa8303b2c5659158d in term 0.
I20260811 22:28:29.463066 22803 leader_election.cc:288] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 05a49b25fe69430aa8303b2c5659158d [CANDIDATE]: Term 1 election: Requested vote from peers 5401f173413b42f8a13e745dcfa51324 (127.29.106.3:34507), 3f852b4ade0e48c5a5b64aa38ed184cb (127.29.106.2:39987)
I20260811 22:28:29.463469 22651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8b9e0f596f4c99b9b8fea3a84fdeac" candidate_uuid: "05a49b25fe69430aa8303b2c5659158d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5401f173413b42f8a13e745dcfa51324"
I20260811 22:28:29.463676 22651 raft_consensus.cc:3037] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 5401f173413b42f8a13e745dcfa51324 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:28:29.463696 22377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8b9e0f596f4c99b9b8fea3a84fdeac" candidate_uuid: "05a49b25fe69430aa8303b2c5659158d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f852b4ade0e48c5a5b64aa38ed184cb"
I20260811 22:28:29.463853 22377 raft_consensus.cc:3037] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 3f852b4ade0e48c5a5b64aa38ed184cb [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:28:29.465826 22651 raft_consensus.cc:2454] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 5401f173413b42f8a13e745dcfa51324 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05a49b25fe69430aa8303b2c5659158d in term 1.
I20260811 22:28:29.465929 22377 raft_consensus.cc:2454] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 3f852b4ade0e48c5a5b64aa38ed184cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05a49b25fe69430aa8303b2c5659158d in term 1.
I20260811 22:28:29.466177 21985 leader_election.cc:302] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 05a49b25fe69430aa8303b2c5659158d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05a49b25fe69430aa8303b2c5659158d, 5401f173413b42f8a13e745dcfa51324; no voters:
I20260811 22:28:29.466449 22803 raft_consensus.cc:2781] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 05a49b25fe69430aa8303b2c5659158d [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:28:29.466881 22803 raft_consensus.cc:686] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 05a49b25fe69430aa8303b2c5659158d [term 1 LEADER]: Becoming Leader. State: Replica: 05a49b25fe69430aa8303b2c5659158d, State: Running, Role: LEADER
I20260811 22:28:29.467342 22803 consensus_queue.cc:237] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 05a49b25fe69430aa8303b2c5659158d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5401f173413b42f8a13e745dcfa51324" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 34507 } } peers { permanent_uuid: "05a49b25fe69430aa8303b2c5659158d" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 36207 } } peers { permanent_uuid: "3f852b4ade0e48c5a5b64aa38ed184cb" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39987 } }
I20260811 22:28:29.471630 21779 catalog_manager.cc:5495] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 05a49b25fe69430aa8303b2c5659158d reported cstate change: term changed from 0 to 1, leader changed from <none> to 05a49b25fe69430aa8303b2c5659158d (127.29.106.1). New cstate: current_term: 1 leader_uuid: "05a49b25fe69430aa8303b2c5659158d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5401f173413b42f8a13e745dcfa51324" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 34507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05a49b25fe69430aa8303b2c5659158d" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 36207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f852b4ade0e48c5a5b64aa38ed184cb" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39987 } health_report { overall_health: UNKNOWN } } }
I20260811 22:28:29.488384 22789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:29.483407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60696 (local address 127.29.106.1:36207)
0811 22:28:29.483563 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:29.483567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:29.483583 (+ 16us) server_negotiation.cc:407] Connection header received
0811 22:28:29.483752 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:29.483755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:29.483824 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:29.483934 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:28:29.484231 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:29.484985 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:29.486488 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:29.486871 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:29.486960 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:29.487462 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:28:29.487900 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:28:29.488007 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:29.488097 (+ 90us) server_negotiation.cc:299] Negotiation successful
0811 22:28:29.488157 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260811 22:28:29.498131 22377 raft_consensus.cc:1261] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 3f852b4ade0e48c5a5b64aa38ed184cb [term 1 FOLLOWER]: Refusing update from remote peer 05a49b25fe69430aa8303b2c5659158d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:28:29.498488 22651 raft_consensus.cc:1261] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 5401f173413b42f8a13e745dcfa51324 [term 1 FOLLOWER]: Refusing update from remote peer 05a49b25fe69430aa8303b2c5659158d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:28:29.499344 22812 consensus_queue.cc:1035] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 05a49b25fe69430aa8303b2c5659158d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f852b4ade0e48c5a5b64aa38ed184cb" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:28:29.499684 22803 consensus_queue.cc:1035] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 05a49b25fe69430aa8303b2c5659158d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5401f173413b42f8a13e745dcfa51324" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 34507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260811 22:28:29.514036 22815 mvcc.cc:205] Tried to move back new op lower bound from 7317452019689725952 to 7317452019581542400. Current Snapshot: MvccSnapshot[applied={T|T < 7317452019689725952 or (T in {7317452019689725952})}]
I20260811 22:28:29.524611 21779 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:28:29.524904 21779 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:28:29.531527 21779 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dbc85509642145079cb9f19ed04547ff: Sending DeleteTablet for 3 replicas of tablet 3d8b9e0f596f4c99b9b8fea3a84fdeac
I20260811 22:28:29.532688 22614 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d8b9e0f596f4c99b9b8fea3a84fdeac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:28:29 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:57890
I20260811 22:28:29.533259 22073 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d8b9e0f596f4c99b9b8fea3a84fdeac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:28:29 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:60692
I20260811 22:28:29.533424 22346 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d8b9e0f596f4c99b9b8fea3a84fdeac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:28:29 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:46726
I20260811 22:28:29.534304 22848 tablet_replica.cc:330] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 3f852b4ade0e48c5a5b64aa38ed184cb: stopping tablet replica
I20260811 22:28:29.534746 22848 raft_consensus.cc:2229] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 3f852b4ade0e48c5a5b64aa38ed184cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:28:29.535245 22848 raft_consensus.cc:2258] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 3f852b4ade0e48c5a5b64aa38ed184cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:28:29.537971 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 21854
I20260811 22:28:29.539785 22848 ts_tablet_manager.cc:1905] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 3f852b4ade0e48c5a5b64aa38ed184cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:28:29.544773 22845 tablet_replica.cc:330] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 5401f173413b42f8a13e745dcfa51324: stopping tablet replica
I20260811 22:28:29.546895 22848 ts_tablet_manager.cc:1918] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 3f852b4ade0e48c5a5b64aa38ed184cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:28:29.547048 22848 log.cc:1196] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 3f852b4ade0e48c5a5b64aa38ed184cb: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-1/wal/wals/3d8b9e0f596f4c99b9b8fea3a84fdeac
I20260811 22:28:29.547551 22845 raft_consensus.cc:2229] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 5401f173413b42f8a13e745dcfa51324 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:28:29.547627 22848 ts_tablet_manager.cc:1939] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 3f852b4ade0e48c5a5b64aa38ed184cb: Deleting consensus metadata
I20260811 22:28:29.548097 22847 tablet_replica.cc:330] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 05a49b25fe69430aa8303b2c5659158d: stopping tablet replica
I20260811 22:28:29.548441 22847 raft_consensus.cc:2229] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 05a49b25fe69430aa8303b2c5659158d [term 1 LEADER]: Raft consensus shutting down.
I20260811 22:28:29.549088 22847 raft_consensus.cc:2258] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 05a49b25fe69430aa8303b2c5659158d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:28:29.549219 22845 raft_consensus.cc:2258] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 5401f173413b42f8a13e745dcfa51324 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:28:29.549283 21757 catalog_manager.cc:4841] TS 3f852b4ade0e48c5a5b64aa38ed184cb (127.29.106.2:39987): tablet 3d8b9e0f596f4c99b9b8fea3a84fdeac (table test-table [id=1ccb48331e4e4370ab29848e7b1c9210]) successfully deleted
I20260811 22:28:29.551110 22847 ts_tablet_manager.cc:1905] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 05a49b25fe69430aa8303b2c5659158d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:28:29.552480 22845 ts_tablet_manager.cc:1905] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 5401f173413b42f8a13e745dcfa51324: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:28:29.562760 22789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:29.538691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60698 (local address 127.29.106.1:36207)
0811 22:28:29.538857 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:29.538863 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:29.539009 (+ 146us) server_negotiation.cc:407] Connection header received
0811 22:28:29.539192 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:29.539197 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:29.539310 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:29.539416 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:28:29.544928 (+ 5512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:29.546192 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:29.548019 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:29.548668 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:29.548789 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:29.553310 (+ 4521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:28:29.553322 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:28:29.553338 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:28:29.553392 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:28:29.559449 (+ 6057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:29.560340 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:29.560350 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:29.560354 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:29.560555 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:29.561012 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:29.561018 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:29.561021 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:29.561486 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:28:29.561803 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:29.562137 (+ 334us) server_negotiation.cc:299] Negotiation successful
0811 22:28:29.562382 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260811 22:28:29.563709 22186 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:28:29.569289 22845 ts_tablet_manager.cc:1918] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 5401f173413b42f8a13e745dcfa51324: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:28:29.569432 22845 log.cc:1196] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 5401f173413b42f8a13e745dcfa51324: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786487105757681-30120-0/minicluster-data/ts-2/wal/wals/3d8b9e0f596f4c99b9b8fea3a84fdeac
I20260811 22:28:29.569861 22845 ts_tablet_manager.cc:1939] T 3d8b9e0f596f4c99b9b8fea3a84fdeac P 5401f173413b42f8a13e745dcfa51324: Deleting consensus metadata
I20260811 22:28:29.571410 21757 catalog_manager.cc:4841] TS 5401f173413b42f8a13e745dcfa51324 (127.29.106.3:34507): tablet 3d8b9e0f596f4c99b9b8fea3a84fdeac (table test-table [id=1ccb48331e4e4370ab29848e7b1c9210]) successfully deleted
I20260811 22:28:29.658565 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 21854
W20260811 22:28:29.680646 21755 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.106.1:36207 (error 108)
I20260811 22:28:29.680862 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 22208
I20260811 22:28:29.682567 22785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:29.682069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:36207 (local address 127.0.0.1:50628)
0811 22:28:29.682242 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:29.682357 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.1:36207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260811 22:28:29.682813 21755 catalog_manager.cc:4584] TS 05a49b25fe69430aa8303b2c5659158d (127.29.106.1:36207): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3d8b9e0f596f4c99b9b8fea3a84fdeac: Network error: Client connection negotiation failed: client connection to 127.29.106.1:36207: connect: Connection refused (error 111)
I20260811 22:28:29.693751 22793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:29.681717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38968 (local address 127.29.106.2:39987)
0811 22:28:29.681936 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:29.681942 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:29.682028 (+ 86us) server_negotiation.cc:407] Connection header received
0811 22:28:29.682250 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:29.682256 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:29.682341 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:29.682461 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:28:29.683995 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:29.684743 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:29.685897 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:29.686281 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:29.686365 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:29.688916 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:28:29.688929 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:28:29.688944 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:28:29.688974 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:28:29.691585 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:29.692175 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:29.692181 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:29.692184 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:29.692321 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:29.692598 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:29.692601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:29.692603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:29.692939 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:28:29.693156 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:29.693362 (+ 206us) server_negotiation.cc:299] Negotiation successful
0811 22:28:29.693514 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260811 22:28:29.694411 22467 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:28:29.736379 22785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:29.735952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:36207 (local address 127.0.0.1:50638)
0811 22:28:29.736154 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:29.736245 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.1:36207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260811 22:28:29.737432 22785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:29.737079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:36207 (local address 127.0.0.1:50644)
0811 22:28:29.737215 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:29.737287 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.1:36207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260811 22:28:29.782100 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 22208
I20260811 22:28:29.797901 22785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:29.797392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:36207 (local address 127.0.0.1:50650)
0811 22:28:29.797610 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:29.797722 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.1:36207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260811 22:28:29.798957 22785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:29.798559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:36207 (local address 127.0.0.1:50660)
0811 22:28:29.798702 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:29.798787 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.1:36207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260811 22:28:29.807461 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 22484
I20260811 22:28:29.820868 22781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:29.808192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37388 (local address 127.29.106.3:34507)
0811 22:28:29.808371 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:29.808376 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:29.808395 (+ 19us) server_negotiation.cc:407] Connection header received
0811 22:28:29.808535 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:29.808540 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:29.808620 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:29.808701 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:28:29.810042 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:29.810795 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:29.812126 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:29.812521 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:29.812634 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:29.815059 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:28:29.815075 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:28:29.815092 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:28:29.815123 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:28:29.818144 (+ 3021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:29.818825 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:29.818833 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:29.818835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:29.818961 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:29.819411 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:29.819415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:29.819417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:29.819770 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:28:29.820073 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:29.820346 (+ 273us) server_negotiation.cc:299] Negotiation successful
0811 22:28:29.820554 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260811 22:28:29.821447 22727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:28:29.879434 22785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:29.878869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:36207 (local address 127.0.0.1:50664)
0811 22:28:29.879124 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:29.879220 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.1:36207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260811 22:28:29.880441 22785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:29.880070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:36207 (local address 127.0.0.1:50680)
0811 22:28:29.880254 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:29.880334 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.1:36207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260811 22:28:29.908507 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 22484
I20260811 22:28:29.935622 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 21716
I20260811 22:28:29.936753 21841 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:28:30.048029 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 21716
I20260811 22:28:30.082003 30120 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260811 22:28:30.426250 30120 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260811 22:28:30.426450 30120 mini_ranger.cc:66] Stopping Ranger...
I20260811 22:28:30.780558 30120 mini_ranger.cc:68] Stopped Ranger
2026-08-11 22:28:30.780 UTC [19951] LOG: received smart shutdown request
2026-08-11 22:28:30.784 UTC [19951] LOG: background worker "logical replication launcher" (PID 19965) exited with exit code 1
2026-08-11 22:28:30.784 UTC [19957] LOG: shutting down
2026-08-11 22:28:30.917 UTC [19951] LOG: database system is shut down
2026-08-11T22:28:30Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50240 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:28:31 dist-test-slave-gjrz krb5kdc[22876](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:28:31 dist-test-slave-gjrz krb5kdc[22876](info): set up 2 sockets
Aug 11 22:28:31 dist-test-slave-gjrz krb5kdc[22876](info): commencing operation
krb5kdc: starting...
W20260811 22:28:32.284029 30120 mini_kdc.cc:121] Time spent starting KDC: real 1.214s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Aug 11 22:28:32 dist-test-slave-gjrz krb5kdc[22876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:28:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:28:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.106.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
I20260811 22:28:32.518855 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:39669
--webserver_interface=127.29.106.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:45579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:39669,127.29.106.61:39349,127.29.106.60:33721
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:28:32.845147 22893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:28:32.845409 22893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:28:32.845539 22893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:28:32.852519 22893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:28:32.852627 22893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:28:32.852722 22893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:28:32.852787 22893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:28:32.852831 22893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:28:32.862186 22893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:45579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:39669,127.29.106.61:39349,127.29.106.60:33721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:39669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.106.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:28:32.863356 22893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:28:32.864593 22893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:28:32.876153 22905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:28:32.876132 22910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:28:32.876092 22907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:28:32.876938 22893 server_base.cc:1034] running on GCE node
I20260811 22:28:32.877743 22893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:28:32.879053 22893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:28:32.880247 22893 hybrid_clock.cc:648] HybridClock initialized: now 1786487312880183 us; error 56 us; skew 500 ppm
I20260811 22:28:32.880584 22893 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:28:32 dist-test-slave-gjrz krb5kdc[22876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487312, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:28:32.883622 22893 init.cc:377] Logged in from keytab as kudu/127.29.106.62@KRBTEST.COM (short username kudu)
I20260811 22:28:32.885025 22893 webserver.cc:460] Webserver started at http://127.29.106.62:38989/ using document root <none> and password file <none>
I20260811 22:28:32.885676 22893 fs_manager.cc:359] Metadata directory not provided
I20260811 22:28:32.885793 22893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:28:32.886086 22893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:28:32.889006 22893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "601062d9e6af4182b94b5094343e95df"
format_stamp: "Formatted at 2026-08-11 22:28:32 on dist-test-slave-gjrz"
server_key: "133c566840f329052806c8cb640a9128"
server_key_iv: "ceec9c23afb8c71d39bce9d461ab414d"
server_key_version: "encryptionkey@0"
I20260811 22:28:32.889776 22893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "601062d9e6af4182b94b5094343e95df"
format_stamp: "Formatted at 2026-08-11 22:28:32 on dist-test-slave-gjrz"
server_key: "133c566840f329052806c8cb640a9128"
server_key_iv: "ceec9c23afb8c71d39bce9d461ab414d"
server_key_version: "encryptionkey@0"
I20260811 22:28:32.894721 22893 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260811 22:28:32.898242 22922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:28:32.899587 22893 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:28:32.899729 22893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "601062d9e6af4182b94b5094343e95df"
format_stamp: "Formatted at 2026-08-11 22:28:32 on dist-test-slave-gjrz"
server_key: "133c566840f329052806c8cb640a9128"
server_key_iv: "ceec9c23afb8c71d39bce9d461ab414d"
server_key_version: "encryptionkey@0"
I20260811 22:28:32.899884 22893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:28:32.912858 22893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:28:32.916638 22893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:28:32.916864 22893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:28:32.934422 22893 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.62:39669
I20260811 22:28:32.934468 23010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.62:39669 every 8 connection(s)
I20260811 22:28:32.935766 22893 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:28:32.939381 23011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:28:32.941613 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 22893
I20260811 22:28:32.941829 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:28:32.942461 30120 external_mini_cluster.cc:1428] Setting key 39167c426ad9032f022ce2e14e20bb02
I20260811 22:28:32.945914 23011 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.106.62" port: 39669 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 22:28:32 dist-test-slave-gjrz krb5kdc[22876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487312, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:28:32.967818 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:32.948978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:39669 (local address 127.0.0.1:50084)
0811 22:28:32.950061 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:32.950101 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0811 22:28:32.951826 (+ 1725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:28:32.952596 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:28:32.952602 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:28:32.953283 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:28:32.953608 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:32.953635 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:32.955174 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:32.955180 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:32.956768 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:32.956783 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:32.957714 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:32.957721 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:32.957881 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:32.958533 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:28:32.958562 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:28:32.960904 (+ 2342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:28:32.963924 (+ 3020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:28:32.963935 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:28:32.963946 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:28:32.964325 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:28:32.964715 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:28:32.964720 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:28:32.964724 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:28:32.964877 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:28:32.965564 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:28:32.965571 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:28:32.965774 (+ 203us) client_negotiation.cc:769] Sending connection context
0811 22:28:32.965970 (+ 196us) client_negotiation.cc:240] Negotiation successful
0811 22:28:32.966580 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":880,"mutex_wait_us":141,"thread_start_us":690,"threads_started":1}
I20260811 22:28:32.967839 23013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:32.948977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50084 (local address 127.29.106.62:39669)
0811 22:28:32.949758 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:32.949772 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:32.950212 (+ 440us) server_negotiation.cc:407] Connection header received
0811 22:28:32.952166 (+ 1954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:32.952171 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:32.952377 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:32.952554 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:28:32.953821 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:32.955025 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:32.956912 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:32.957471 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:32.957606 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:32.961130 (+ 3524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:28:32.961138 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:28:32.961148 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:28:32.961177 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:28:32.963808 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:32.964453 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:32.964459 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:32.964462 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:32.964610 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:32.964986 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:32.964990 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:32.964994 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:32.965436 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:28:32.965629 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:32.966137 (+ 508us) server_negotiation.cc:299] Negotiation successful
0811 22:28:32.966453 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":558,"thread_start_us":357,"threads_started":1}
WARNING: no policy specified for kudu/127.29.106.61@KRBTEST.COM; defaulting to no policy
I20260811 22:28:32.970600 23011 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.106.61" port: 39349 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.61@KRBTEST.COM" created.
I20260811 22:28:32.971743 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:32.971392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.61:39349 (local address 127.0.0.1:56208)
0811 22:28:32.971544 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:32.971646 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.61:39349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260811 22:28:32.972044 22927 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.106.61:39349: connect: Connection refused (error 111)
I20260811 22:28:32.973462 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:32.973159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.61:39349 (local address 127.0.0.1:56210)
0811 22:28:32.973285 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:32.973350 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.61:39349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260811 22:28:32.973641 23011 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.106.61:39349: Network error: Client connection negotiation failed: client connection to 127.29.106.61:39349: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.61.keytab.
Entry for principal kudu/127.29.106.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.61.keytab.
I20260811 22:28:32.998119 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.61:39349
--webserver_interface=127.29.106.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.61.keytab
--principal=kudu/127.29.106.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:45579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:39669,127.29.106.61:39349,127.29.106.60:33721
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 22:28:33.022938 23011 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.106.61" port: 39349 } attempt: 1
I20260811 22:28:33.024155 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:33.023645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.61:39349 (local address 127.0.0.1:56222)
0811 22:28:33.023850 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:33.023986 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.61:39349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260811 22:28:33.025166 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:33.024764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.61:39349 (local address 127.0.0.1:56228)
0811 22:28:33.024918 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:33.025011 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.61:39349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260811 22:28:33.025339 23011 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.106.61:39349: Network error: Client connection negotiation failed: client connection to 127.29.106.61:39349: connect: Connection refused (error 111)
I20260811 22:28:33.093588 23011 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.106.61" port: 39349 } attempt: 2
I20260811 22:28:33.094894 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:33.094339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.61:39349 (local address 127.0.0.1:56236)
0811 22:28:33.094611 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:33.094723 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.61:39349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260811 22:28:33.095956 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:33.095554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.61:39349 (local address 127.0.0.1:56252)
0811 22:28:33.095700 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:33.095799 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.61:39349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260811 22:28:33.096200 23011 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.106.61:39349: Network error: Client connection negotiation failed: client connection to 127.29.106.61:39349: connect: Connection refused (error 111)
I20260811 22:28:33.187475 23011 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.106.61" port: 39349 } attempt: 3
I20260811 22:28:33.188982 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:33.188362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.61:39349 (local address 127.0.0.1:56258)
0811 22:28:33.188639 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:33.188788 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.61:39349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260811 22:28:33.190155 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:33.189710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.61:39349 (local address 127.0.0.1:56270)
0811 22:28:33.189876 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:33.189973 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.61:39349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260811 22:28:33.190379 23011 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.106.61:39349: Network error: Client connection negotiation failed: client connection to 127.29.106.61:39349: connect: Connection refused (error 111)
W20260811 22:28:33.323614 23024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:28:33.323871 23024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:28:33.323972 23024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:28:33.330879 23024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:28:33.330976 23024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:28:33.331027 23024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:28:33.331082 23024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:28:33.331125 23024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:28:33.333648 23011 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.106.61" port: 39349 } attempt: 4
I20260811 22:28:33.334878 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:33.334278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.61:39349 (local address 127.0.0.1:56278)
0811 22:28:33.334606 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:33.334713 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.61:39349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
I20260811 22:28:33.335923 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:33.335531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.61:39349 (local address 127.0.0.1:56282)
0811 22:28:33.335671 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:33.335777 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.61:39349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260811 22:28:33.336161 23011 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.106.61:39349: Network error: Client connection negotiation failed: client connection to 127.29.106.61:39349: connect: Connection refused (error 111)
I20260811 22:28:33.341162 23024 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:45579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:39669,127.29.106.61:39349,127.29.106.60:33721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.61.keytab
--principal=kudu/127.29.106.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.61:39349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.29.106.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:28:33.342375 23024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:28:33.343636 23024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:28:33.354786 23035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:28:33.354797 23038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:28:33.355010 23034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:28:33.355840 23024 server_base.cc:1034] running on GCE node
I20260811 22:28:33.356537 23024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:28:33.357777 23024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:28:33.358976 23024 hybrid_clock.cc:648] HybridClock initialized: now 1786487313358938 us; error 25 us; skew 500 ppm
I20260811 22:28:33.359323 23024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:28:33 dist-test-slave-gjrz krb5kdc[22876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487313, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:28:33.369529 23024 init.cc:377] Logged in from keytab as kudu/127.29.106.61@KRBTEST.COM (short username kudu)
I20260811 22:28:33.370915 23024 webserver.cc:460] Webserver started at http://127.29.106.61:33813/ using document root <none> and password file <none>
I20260811 22:28:33.371584 23024 fs_manager.cc:359] Metadata directory not provided
I20260811 22:28:33.371696 23024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:28:33.371989 23024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:28:33.374825 23024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/minicluster-data/master-1/data/instance:
uuid: "600774ae65ac4617af9542f8549ca7b3"
format_stamp: "Formatted at 2026-08-11 22:28:33 on dist-test-slave-gjrz"
server_key: "e5544d5d9a212bcec5856ecf12987a04"
server_key_iv: "cd3c81d22da1b2df548fa75bb4021e5a"
server_key_version: "encryptionkey@0"
I20260811 22:28:33.375687 23024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/minicluster-data/master-1/wal/instance:
uuid: "600774ae65ac4617af9542f8549ca7b3"
format_stamp: "Formatted at 2026-08-11 22:28:33 on dist-test-slave-gjrz"
server_key: "e5544d5d9a212bcec5856ecf12987a04"
server_key_iv: "cd3c81d22da1b2df548fa75bb4021e5a"
server_key_version: "encryptionkey@0"
I20260811 22:28:33.380566 23024 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260811 22:28:33.383803 23059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:28:33.384999 23024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 22:28:33.385156 23024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/minicluster-data/master-1/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/minicluster-data/master-1/wal
uuid: "600774ae65ac4617af9542f8549ca7b3"
format_stamp: "Formatted at 2026-08-11 22:28:33 on dist-test-slave-gjrz"
server_key: "e5544d5d9a212bcec5856ecf12987a04"
server_key_iv: "cd3c81d22da1b2df548fa75bb4021e5a"
server_key_version: "encryptionkey@0"
I20260811 22:28:33.385320 23024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:28:33.407859 23024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:28:33.411593 23024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:28:33.411834 23024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:28:33.428745 23024 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.61:39349
I20260811 22:28:33.428807 23157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.61:39349 every 8 connection(s)
I20260811 22:28:33.430009 23024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/minicluster-data/master-1/data/info.pb
I20260811 22:28:33.433516 23158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:28:33.439695 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 23024
I20260811 22:28:33.439857 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/minicluster-data/master-1/wal/instance
I20260811 22:28:33.440270 30120 external_mini_cluster.cc:1428] Setting key cf7e6777b00b01e4efaf44e538b2502e
I20260811 22:28:33.439826 23158 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.106.62" port: 39669 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 22:28:33 dist-test-slave-gjrz krb5kdc[22876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487313, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.61@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:28:33.459583 23013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:33.442332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50086 (local address 127.29.106.62:39669)
0811 22:28:33.442489 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:33.442493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:33.443877 (+ 1384us) server_negotiation.cc:407] Connection header received
0811 22:28:33.445669 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:33.445675 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:33.445755 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:33.445846 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:28:33.447337 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:33.448010 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:33.449515 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:33.449945 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:33.450056 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:33.453980 (+ 3924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:28:33.453985 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:28:33.453988 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:28:33.454022 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:28:33.456585 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:33.457401 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:33.457404 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:33.457406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:33.457518 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:33.457969 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:33.457972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:33.457974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:33.458244 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:28:33.458359 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:33.459146 (+ 787us) server_negotiation.cc:299] Negotiation successful
0811 22:28:33.459350 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260811 22:28:33.460837 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:33.442950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:39669 (local address 127.0.0.1:50086)
0811 22:28:33.443716 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:33.443753 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0811 22:28:33.445390 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:28:33.446096 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:28:33.446104 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:28:33.446592 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:28:33.447127 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:33.447145 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:33.448175 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:33.448178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:33.449363 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:33.449376 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:33.450171 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:33.450178 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:33.450395 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:33.451045 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:28:33.451080 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:28:33.453762 (+ 2682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:28:33.456728 (+ 2966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:28:33.456736 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:28:33.456749 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:28:33.457307 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:28:33.457629 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:28:33.457634 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:28:33.457637 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:28:33.457875 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:28:33.458367 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:28:33.458374 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:28:33.458767 (+ 393us) client_negotiation.cc:769] Sending connection context
0811 22:28:33.459093 (+ 326us) client_negotiation.cc:240] Negotiation successful
0811 22:28:33.459580 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":319,"threads_started":1}
I20260811 22:28:33.462529 23158 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.106.61" port: 39349 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.29.106.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.60@KRBTEST.COM" created.
Aug 11 22:28:33 dist-test-slave-gjrz krb5kdc[22876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487313, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.61@KRBTEST.COM for kudu/127.29.106.61@KRBTEST.COM
I20260811 22:28:33.476923 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:33.463348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.61:39349 (local address 127.0.0.1:56286)
0811 22:28:33.463514 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:33.463536 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 22:28:33.463631 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:28:33.464666 (+ 1035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:28:33.464669 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:28:33.464991 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:28:33.465136 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:33.465144 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:33.467062 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:33.467068 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:33.467939 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:33.467954 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:33.468698 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:33.468702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:33.468842 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:33.469331 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:28:33.469346 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:28:33.471539 (+ 2193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:28:33.474424 (+ 2885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:28:33.474428 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:28:33.474432 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:28:33.474888 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:28:33.475319 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:28:33.475323 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:28:33.475326 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:28:33.475478 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:28:33.476057 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:28:33.476061 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:28:33.476190 (+ 129us) client_negotiation.cc:769] Sending connection context
0811 22:28:33.476319 (+ 129us) client_negotiation.cc:240] Negotiation successful
0811 22:28:33.476582 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260811 22:28:33.476986 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:33.463462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56286 (local address 127.29.106.61:39349)
0811 22:28:33.464193 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:33.464202 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:33.464233 (+ 31us) server_negotiation.cc:407] Connection header received
0811 22:28:33.464287 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:33.464295 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:33.464518 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:33.464664 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:28:33.465290 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:33.466908 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:33.468069 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:33.468575 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:33.468667 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:33.471728 (+ 3061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:28:33.471740 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:28:33.471752 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:28:33.471784 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:28:33.474291 (+ 2507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:33.475036 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:33.475042 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:33.475045 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:33.475177 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:33.475595 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:33.475599 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:33.475601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:33.475918 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:28:33.476095 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:33.476443 (+ 348us) server_negotiation.cc:299] Negotiation successful
0811 22:28:33.476687 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":553,"thread_start_us":406,"threads_started":1}
I20260811 22:28:33.479179 23158 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.106.60" port: 33721 } has no permanent_uuid. Determining permanent_uuid...
I20260811 22:28:33.480201 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:33.479784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:33721 (local address 127.0.0.1:48432)
0811 22:28:33.479933 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:33.480059 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:33721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260811 22:28:33.480443 23066 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.106.60:33721: connect: Connection refused (error 111)
I20260811 22:28:33.481812 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:33.481523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:33721 (local address 127.0.0.1:48442)
0811 22:28:33.481648 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:33.481723 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:33721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260811 22:28:33.482029 23158 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.106.60:33721: Network error: Client connection negotiation failed: client connection to 127.29.106.60:33721: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.60.keytab.
Entry for principal kudu/127.29.106.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.60.keytab.
I20260811 22:28:33.504330 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.60:33721
--webserver_interface=127.29.106.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.60.keytab
--principal=kudu/127.29.106.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:45579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:39669,127.29.106.61:39349,127.29.106.60:33721
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 22:28:33.531349 23158 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.106.60" port: 33721 } attempt: 1
I20260811 22:28:33.532480 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:33.532041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:33721 (local address 127.0.0.1:48458)
0811 22:28:33.532232 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:33.532338 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:33721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260811 22:28:33.533394 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:33.533077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:33721 (local address 127.0.0.1:48460)
0811 22:28:33.533204 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:33.533286 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:33721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260811 22:28:33.533592 23158 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.106.60:33721: Network error: Client connection negotiation failed: client connection to 127.29.106.60:33721: connect: Connection refused (error 111)
I20260811 22:28:33.601800 23158 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.106.60" port: 33721 } attempt: 2
I20260811 22:28:33.603003 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:33.602491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:33721 (local address 127.0.0.1:48464)
0811 22:28:33.602710 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:33.602817 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:33721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260811 22:28:33.604067 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:33.603693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:33721 (local address 127.0.0.1:48478)
0811 22:28:33.603851 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:33.603952 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:33721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260811 22:28:33.604266 23158 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.106.60:33721: Network error: Client connection negotiation failed: client connection to 127.29.106.60:33721: connect: Connection refused (error 111)
I20260811 22:28:33.635424 23011 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.106.61" port: 39349 } attempt: 5
Aug 11 22:28:33 dist-test-slave-gjrz krb5kdc[22876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487312, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for kudu/127.29.106.61@KRBTEST.COM
I20260811 22:28:33.649525 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:33.636243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56288 (local address 127.29.106.61:39349)
0811 22:28:33.636415 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:33.636420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:33.636440 (+ 20us) server_negotiation.cc:407] Connection header received
0811 22:28:33.636634 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:33.636638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:33.636703 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:33.636787 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:28:33.637869 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:33.638582 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:33.640385 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:33.640846 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:33.640943 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:33.644540 (+ 3597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:28:33.644545 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:28:33.644548 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:28:33.644582 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:28:33.647056 (+ 2474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:33.647709 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:33.647714 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:33.647716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:33.647834 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:33.648198 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:33.648203 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:33.648205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:33.648484 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:28:33.648619 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:33.649018 (+ 399us) server_negotiation.cc:299] Negotiation successful
0811 22:28:33.649225 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260811 22:28:33.649519 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:33.636115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.61:39349 (local address 127.0.0.1:56288)
0811 22:28:33.636339 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:33.636363 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0811 22:28:33.636499 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:28:33.636852 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:28:33.636857 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:28:33.637446 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:28:33.637722 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:33.637738 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:33.638726 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:33.638732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:33.640212 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:33.640229 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:33.640999 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:33.641006 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:33.641208 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:33.641801 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:28:33.641824 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:28:33.644345 (+ 2521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:28:33.647186 (+ 2841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:28:33.647191 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:28:33.647195 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:28:33.647590 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:28:33.647942 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:28:33.647948 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:28:33.647952 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:28:33.648086 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:28:33.648602 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:28:33.648607 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:28:33.648751 (+ 144us) client_negotiation.cc:769] Sending connection context
0811 22:28:33.648863 (+ 112us) client_negotiation.cc:240] Negotiation successful
0811 22:28:33.649155 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260811 22:28:33.650477 23011 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.106.60" port: 33721 } has no permanent_uuid. Determining permanent_uuid...
I20260811 22:28:33.651528 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:33.651133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:33721 (local address 127.0.0.1:48492)
0811 22:28:33.651314 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:33.651387 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:33721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260811 22:28:33.652467 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:33.652158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:33721 (local address 127.0.0.1:48500)
0811 22:28:33.652266 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:33.652343 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:33721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260811 22:28:33.652652 23011 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.106.60:33721: Network error: Client connection negotiation failed: client connection to 127.29.106.60:33721: connect: Connection refused (error 111)
I20260811 22:28:33.695449 23158 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.106.60" port: 33721 } attempt: 3
I20260811 22:28:33.696590 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:33.696105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:33721 (local address 127.0.0.1:48504)
0811 22:28:33.696332 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:33.696419 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:33721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260811 22:28:33.697691 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:33.697252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:33721 (local address 127.0.0.1:48514)
0811 22:28:33.697490 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:33.697575 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:33721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
W20260811 22:28:33.697904 23158 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.106.60:33721: Network error: Client connection negotiation failed: client connection to 127.29.106.60:33721: connect: Connection refused (error 111)
I20260811 22:28:33.703826 23011 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.106.60" port: 33721 } attempt: 1
I20260811 22:28:33.704758 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:33.704348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:33721 (local address 127.0.0.1:48526)
0811 22:28:33.704511 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:33.704615 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:33721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260811 22:28:33.705605 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:33.705327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:33721 (local address 127.0.0.1:48542)
0811 22:28:33.705439 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:33.705499 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:33721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260811 22:28:33.705792 23011 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.106.60:33721: Network error: Client connection negotiation failed: client connection to 127.29.106.60:33721: connect: Connection refused (error 111)
I20260811 22:28:33.773980 23011 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.106.60" port: 33721 } attempt: 2
I20260811 22:28:33.775128 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:33.774660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:33721 (local address 127.0.0.1:48548)
0811 22:28:33.774868 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:33.774967 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:33721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260811 22:28:33.776122 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:33.775819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:33721 (local address 127.0.0.1:48564)
0811 22:28:33.775933 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:33.775999 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:33721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260811 22:28:33.776320 23011 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.106.60:33721: Network error: Client connection negotiation failed: client connection to 127.29.106.60:33721: connect: Connection refused (error 111)
W20260811 22:28:33.831828 23173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:28:33.832084 23173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:28:33.832202 23173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:28:33.839170 23173 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:28:33.839278 23173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:28:33.839354 23173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:28:33.839430 23173 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:28:33.839493 23173 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:28:33.841097 23158 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.106.60" port: 33721 } attempt: 4
I20260811 22:28:33.842227 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:33.841767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:33721 (local address 127.0.0.1:48568)
0811 22:28:33.841991 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:33.842081 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:33721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260811 22:28:33.843124 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:33.842829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:33721 (local address 127.0.0.1:48578)
0811 22:28:33.842955 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:33.843022 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:33721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260811 22:28:33.843319 23158 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.106.60:33721: Network error: Client connection negotiation failed: client connection to 127.29.106.60:33721: connect: Connection refused (error 111)
I20260811 22:28:33.849283 23173 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:45579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:39669,127.29.106.61:39349,127.29.106.60:33721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.60.keytab
--principal=kudu/127.29.106.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.60:33721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.29.106.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:28:33.850430 23173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:28:33.851686 23173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:28:33.863147 23179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:28:33.863190 23180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:28:33.863225 23182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:28:33.864022 23173 server_base.cc:1034] running on GCE node
I20260811 22:28:33.864698 23173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:28:33.865928 23173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:28:33.867128 23173 hybrid_clock.cc:648] HybridClock initialized: now 1786487313867074 us; error 46 us; skew 500 ppm
I20260811 22:28:33.867544 23173 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:28:33 dist-test-slave-gjrz krb5kdc[22876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487313, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:28:33.870476 23173 init.cc:377] Logged in from keytab as kudu/127.29.106.60@KRBTEST.COM (short username kudu)
I20260811 22:28:33.871948 23173 webserver.cc:460] Webserver started at http://127.29.106.60:38671/ using document root <none> and password file <none>
I20260811 22:28:33.872583 23173 fs_manager.cc:359] Metadata directory not provided
I20260811 22:28:33.872699 23173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:28:33.872977 23173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:28:33.875873 23173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/minicluster-data/master-2/data/instance:
uuid: "32d41cb9920d4283900dd689c05850e0"
format_stamp: "Formatted at 2026-08-11 22:28:33 on dist-test-slave-gjrz"
server_key: "2db67148176fa0c0d7633c181512c07e"
server_key_iv: "61f2c09ee61023f28c186bf4ecdbd6f7"
server_key_version: "encryptionkey@0"
I20260811 22:28:33.876648 23173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/minicluster-data/master-2/wal/instance:
uuid: "32d41cb9920d4283900dd689c05850e0"
format_stamp: "Formatted at 2026-08-11 22:28:33 on dist-test-slave-gjrz"
server_key: "2db67148176fa0c0d7633c181512c07e"
server_key_iv: "61f2c09ee61023f28c186bf4ecdbd6f7"
server_key_version: "encryptionkey@0"
I20260811 22:28:33.881588 23173 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260811 22:28:33.882504 23011 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.106.60" port: 33721 } attempt: 3
I20260811 22:28:33.883612 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:33.883104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:33721 (local address 127.0.0.1:48586)
0811 22:28:33.883360 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:33.883456 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:33721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260811 22:28:33.884572 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:33.884194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:33721 (local address 127.0.0.1:48588)
0811 22:28:33.884388 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:33.884472 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:33721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260811 22:28:33.884819 23011 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.106.60:33721: Network error: Client connection negotiation failed: client connection to 127.29.106.60:33721: connect: Connection refused (error 111)
I20260811 22:28:33.885064 23191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:28:33.886243 23173 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260811 22:28:33.886386 23173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/minicluster-data/master-2/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/minicluster-data/master-2/wal
uuid: "32d41cb9920d4283900dd689c05850e0"
format_stamp: "Formatted at 2026-08-11 22:28:33 on dist-test-slave-gjrz"
server_key: "2db67148176fa0c0d7633c181512c07e"
server_key_iv: "61f2c09ee61023f28c186bf4ecdbd6f7"
server_key_version: "encryptionkey@0"
I20260811 22:28:33.886559 23173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:28:33.908713 23173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:28:33.912626 23173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:28:33.912887 23173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:28:33.930114 23173 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.60:33721
I20260811 22:28:33.930150 23288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.60:33721 every 8 connection(s)
I20260811 22:28:33.931394 23173 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/minicluster-data/master-2/data/info.pb
I20260811 22:28:33.934844 23290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:28:33.936667 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 23173
I20260811 22:28:33.936870 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786487105757681-30120-0/minicluster-data/master-2/wal/instance
I20260811 22:28:33.937454 30120 external_mini_cluster.cc:1428] Setting key 079c5b623d458aeafd4916323f38ea54
I20260811 22:28:33.943992 23290 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.106.62" port: 39669 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 22:28:33 dist-test-slave-gjrz krb5kdc[22876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.60@KRBTEST.COM
Aug 11 22:28:33 dist-test-slave-gjrz krb5kdc[22876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487313, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.60@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:28:33.962971 23013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:33.947707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50088 (local address 127.29.106.62:39669)
0811 22:28:33.947897 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:33.947903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:33.948530 (+ 627us) server_negotiation.cc:407] Connection header received
0811 22:28:33.948706 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:33.948712 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:33.948771 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:33.948849 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:28:33.950077 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:33.950762 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:33.952336 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:33.952839 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:33.952955 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:33.956927 (+ 3972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:28:33.956933 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:28:33.956937 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:28:33.956973 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:28:33.959979 (+ 3006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:33.960582 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:33.960585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:33.960588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:33.960717 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:33.961254 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:33.961259 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:33.961263 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:33.961570 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:28:33.961716 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:33.962380 (+ 664us) server_negotiation.cc:299] Negotiation successful
0811 22:28:33.962624 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260811 22:28:33.963865 23294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:33.947700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:39669 (local address 127.0.0.1:50088)
0811 22:28:33.948409 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:33.948441 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0811 22:28:33.948589 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:28:33.948938 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:28:33.948945 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:28:33.949535 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:28:33.949862 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:33.949872 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:33.950918 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:33.950923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:33.952189 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:33.952204 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:33.953241 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:33.953249 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:33.953429 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:33.953992 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:28:33.954029 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:28:33.956738 (+ 2709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:28:33.960108 (+ 3370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:28:33.960118 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:28:33.960126 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:28:33.960485 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:28:33.960869 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:28:33.960874 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:28:33.960877 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:28:33.961139 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:28:33.961708 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:28:33.961716 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:28:33.962039 (+ 323us) client_negotiation.cc:769] Sending connection context
0811 22:28:33.962271 (+ 232us) client_negotiation.cc:240] Negotiation successful
0811 22:28:33.962698 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":351,"threads_started":1}
I20260811 22:28:33.963903 23293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:33.939427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48600 (local address 127.29.106.60:33721)
0811 22:28:33.943427 (+ 4000us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:33.943442 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:33.943849 (+ 407us) server_negotiation.cc:407] Connection header received
0811 22:28:33.945890 (+ 2041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:33.945911 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:33.946307 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:33.946607 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:28:33.948301 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:33.949798 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:33.951287 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:33.951915 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:33.952078 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:33.956658 (+ 4580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:28:33.956666 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:28:33.956677 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:28:33.956712 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:28:33.959982 (+ 3270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:33.961243 (+ 1261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:33.961251 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:33.961254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:33.961374 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:33.961750 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:33.961753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:33.961756 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:33.962124 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:28:33.962405 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:33.962741 (+ 336us) server_negotiation.cc:299] Negotiation successful
0811 22:28:33.962968 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3720,"thread_start_us":3465,"threads_started":1}
I20260811 22:28:33.965363 23290 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.106.61" port: 39349 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 22:28:33 dist-test-slave-gjrz krb5kdc[22876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487313, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.60@KRBTEST.COM for kudu/127.29.106.61@KRBTEST.COM
Aug 11 22:28:33 dist-test-slave-gjrz krb5kdc[22876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:28:33.977787 23294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:33.966109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.61:39349 (local address 127.0.0.1:56294)
0811 22:28:33.966256 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:33.966273 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 22:28:33.966351 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:28:33.966931 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:28:33.966934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:28:33.967230 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:28:33.967398 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:33.967405 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:33.968317 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:33.968322 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:33.969223 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:33.969236 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:33.969904 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:33.969909 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:33.970084 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:33.970533 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:28:33.970557 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:28:33.972826 (+ 2269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:28:33.975626 (+ 2800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:28:33.975629 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:28:33.975632 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:28:33.975964 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:28:33.976335 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:28:33.976339 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:28:33.976342 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:28:33.976472 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:28:33.976942 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:28:33.976947 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:28:33.977097 (+ 150us) client_negotiation.cc:769] Sending connection context
0811 22:28:33.977217 (+ 120us) client_negotiation.cc:240] Negotiation successful
0811 22:28:33.977465 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260811 22:28:33.977900 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:33.966179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56294 (local address 127.29.106.61:39349)
0811 22:28:33.966357 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:33.966362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:33.966408 (+ 46us) server_negotiation.cc:407] Connection header received
0811 22:28:33.966458 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:33.966461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:33.966519 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:33.966598 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:28:33.967538 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:33.968191 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:33.969353 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:33.969777 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:33.969882 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:33.973001 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:28:33.973006 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:28:33.973009 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:28:33.973053 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:28:33.975513 (+ 2460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:33.976080 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:33.976086 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:33.976089 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:33.976230 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:33.976580 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:33.976584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:33.976586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:33.976830 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:28:33.976960 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:33.977327 (+ 367us) server_negotiation.cc:299] Negotiation successful
0811 22:28:33.977536 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260811 22:28:33.978942 23290 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.106.60" port: 33721 } has no permanent_uuid. Determining permanent_uuid...
I20260811 22:28:33.980108 23013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:33.966819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50096 (local address 127.29.106.62:39669)
0811 22:28:33.966986 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:33.966992 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:33.967007 (+ 15us) server_negotiation.cc:407] Connection header received
0811 22:28:33.967058 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:33.967061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:33.967111 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:33.967187 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:28:33.968359 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:33.969084 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:33.970406 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:33.970796 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:33.970896 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:33.975079 (+ 4183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:28:33.975084 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:28:33.975087 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:28:33.975115 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:28:33.977543 (+ 2428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:33.978213 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:33.978217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:33.978219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:33.978338 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:33.978724 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:33.978729 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:33.978732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:33.979033 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:28:33.979342 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:33.979609 (+ 267us) server_negotiation.cc:299] Negotiation successful
0811 22:28:33.979813 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Aug 11 22:28:33 dist-test-slave-gjrz krb5kdc[22876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487313, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.60@KRBTEST.COM for kudu/127.29.106.60@KRBTEST.COM
Aug 11 22:28:33 dist-test-slave-gjrz krb5kdc[22876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.61@KRBTEST.COM
I20260811 22:28:33.991458 23294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:33.979541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:33721 (local address 127.0.0.1:48602)
0811 22:28:33.979809 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:33.979820 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 22:28:33.979909 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:28:33.980153 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:28:33.980157 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:28:33.980499 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:28:33.980654 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:33.980666 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:33.981638 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:33.981642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:33.982876 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:33.982892 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:33.983595 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:33.983602 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:33.983747 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:33.984216 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:28:33.984232 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:28:33.986252 (+ 2020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:28:33.989098 (+ 2846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:28:33.989103 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:28:33.989107 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:28:33.989464 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:28:33.989851 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:28:33.989856 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:28:33.989859 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:28:33.990019 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:28:33.990598 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:28:33.990602 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:28:33.990750 (+ 148us) client_negotiation.cc:769] Sending connection context
0811 22:28:33.990884 (+ 134us) client_negotiation.cc:240] Negotiation successful
0811 22:28:33.991148 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218}
I20260811 22:28:33.992053 23293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:33.979617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48602 (local address 127.29.106.60:33721)
0811 22:28:33.979747 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:33.979752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:33.979881 (+ 129us) server_negotiation.cc:407] Connection header received
0811 22:28:33.979991 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:33.979996 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:33.980053 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:33.980125 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:28:33.980788 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:33.981491 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:33.983010 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:33.983450 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:33.983538 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:33.986423 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:28:33.986429 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:28:33.986434 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:28:33.986470 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:28:33.988982 (+ 2512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:33.989580 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:33.989585 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:33.989588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:33.989704 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:33.990151 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:33.990155 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:33.990158 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:33.990416 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:28:33.990540 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:33.991705 (+ 1165us) server_negotiation.cc:299] Negotiation successful
0811 22:28:33.991867 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260811 22:28:33.995743 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:33.981662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56300 (local address 127.29.106.61:39349)
0811 22:28:33.981813 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:33.981817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:33.981955 (+ 138us) server_negotiation.cc:407] Connection header received
0811 22:28:33.982136 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:33.982139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:33.982188 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:33.982263 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:28:33.983663 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:33.984461 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:33.985955 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:33.986364 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:33.986482 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:33.990566 (+ 4084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:28:33.990572 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:28:33.990577 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:28:33.990612 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:28:33.993369 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:33.993947 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:33.993952 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:33.993954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:33.994086 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:33.994408 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:33.994411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:33.994413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:33.994670 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:28:33.994917 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:33.995191 (+ 274us) server_negotiation.cc:299] Negotiation successful
0811 22:28:33.995462 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260811 22:28:33.996790 23290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32d41cb9920d4283900dd689c05850e0: Bootstrap starting.
I20260811 22:28:34.000190 30120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:28:34.000604 23290 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32d41cb9920d4283900dd689c05850e0: Neither blocks nor log segments found. Creating new log.
I20260811 22:28:34.001602 23290 log.cc:824] T 00000000000000000000000000000000 P 32d41cb9920d4283900dd689c05850e0: Log is configured to *not* fsync() on all Append() calls
I20260811 22:28:34.004523 23290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32d41cb9920d4283900dd689c05850e0: No bootstrap required, opened a new log
I20260811 22:28:34.015463 23290 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32d41cb9920d4283900dd689c05850e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601062d9e6af4182b94b5094343e95df" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 39669 } } peers { permanent_uuid: "600774ae65ac4617af9542f8549ca7b3" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 39349 } } peers { permanent_uuid: "32d41cb9920d4283900dd689c05850e0" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 33721 } }
I20260811 22:28:34.016758 23290 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32d41cb9920d4283900dd689c05850e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:28:34.016957 23290 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32d41cb9920d4283900dd689c05850e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32d41cb9920d4283900dd689c05850e0, State: Initialized, Role: FOLLOWER
I20260811 22:28:34.017781 23290 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32d41cb9920d4283900dd689c05850e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601062d9e6af4182b94b5094343e95df" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 39669 } } peers { permanent_uuid: "600774ae65ac4617af9542f8549ca7b3" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 39349 } } peers { permanent_uuid: "32d41cb9920d4283900dd689c05850e0" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 33721 } }
I20260811 22:28:34.024823 23308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32d41cb9920d4283900dd689c05850e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601062d9e6af4182b94b5094343e95df" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 39669 } } peers { permanent_uuid: "600774ae65ac4617af9542f8549ca7b3" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 39349 } } peers { permanent_uuid: "32d41cb9920d4283900dd689c05850e0" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 33721 } } }
I20260811 22:28:34.025341 23013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:34.005053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50100 (local address 127.29.106.62:39669)
0811 22:28:34.005398 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:34.005404 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:34.006063 (+ 659us) server_negotiation.cc:407] Connection header received
0811 22:28:34.006270 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:34.006276 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:34.006360 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:34.006451 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:28:34.009376 (+ 2925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:34.010587 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:34.013394 (+ 2807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:34.014030 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:34.014137 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:34.018104 (+ 3967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:28:34.018110 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:28:34.018114 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:28:34.018164 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:28:34.021906 (+ 3742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:34.022789 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:34.022796 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:34.022800 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:34.022982 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:34.023494 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:34.023500 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:34.023504 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:34.023884 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:28:34.024147 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:34.024754 (+ 607us) server_negotiation.cc:299] Negotiation successful
0811 22:28:34.024986 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227}
I20260811 22:28:34.027171 23308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32d41cb9920d4283900dd689c05850e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:28:34.027036 23290 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32d41cb9920d4283900dd689c05850e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:28:34.027670 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:34.005219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56316 (local address 127.29.106.61:39349)
0811 22:28:34.005518 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:34.005523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:34.010398 (+ 4875us) server_negotiation.cc:407] Connection header received
0811 22:28:34.010633 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:34.010639 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:34.010712 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:34.010774 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:28:34.013259 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:34.014405 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:34.016764 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:34.017345 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:34.017443 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:34.020705 (+ 3262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:28:34.020711 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:28:34.020716 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:28:34.020754 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:28:34.024262 (+ 3508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:34.025171 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:34.025177 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:34.025180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:34.025335 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:34.025790 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:34.025795 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:34.025799 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:34.026170 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:28:34.026390 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:34.027068 (+ 678us) server_negotiation.cc:299] Negotiation successful
0811 22:28:34.027325 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203}
I20260811 22:28:34.032398 23293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:34.009761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48614 (local address 127.29.106.60:33721)
0811 22:28:34.009932 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:34.009938 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:34.016495 (+ 6557us) server_negotiation.cc:407] Connection header received
0811 22:28:34.016680 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:34.016686 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:34.016763 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:34.016850 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:28:34.018853 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:34.019970 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:34.021834 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:34.022548 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:34.022645 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:34.025933 (+ 3288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:28:34.025939 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:28:34.025943 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:28:34.025980 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:28:34.029412 (+ 3432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:34.030202 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:34.030207 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:34.030211 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:34.030403 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:34.030706 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:34.030711 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:34.030714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:34.031116 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:28:34.031424 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:34.031733 (+ 309us) server_negotiation.cc:299] Negotiation successful
0811 22:28:34.031986 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260811 22:28:34.039633 23332 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 32d41cb9920d4283900dd689c05850e0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 22:28:34.039757 23332 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 22:28:34.037243 23173 master_runner.cc:186] Error getting master registration for 127.29.106.62:39669: OK, instance_id { permanent_uuid: "601062d9e6af4182b94b5094343e95df" instance_seqno: 1786487312929319 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260811 22:28:34.048262 23173 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260811 22:28:34.062014 23011 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.106.60" port: 33721 } attempt: 4
Aug 11 22:28:34 dist-test-slave-gjrz krb5kdc[22876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487312, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for kudu/127.29.106.60@KRBTEST.COM
I20260811 22:28:34.076148 23293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:34.062583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48628 (local address 127.29.106.60:33721)
0811 22:28:34.062739 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:34.062744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:34.062766 (+ 22us) server_negotiation.cc:407] Connection header received
0811 22:28:34.062946 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:34.062949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:34.063008 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:34.063088 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:28:34.064066 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:34.064835 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:34.066614 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:34.067048 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:34.067150 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:34.070975 (+ 3825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:28:34.070980 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:28:34.070983 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:28:34.071013 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:28:34.073648 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:34.074383 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:34.074387 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:34.074389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:34.074524 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:34.074889 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:34.074892 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:34.074894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:34.075171 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:28:34.075314 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:34.075708 (+ 394us) server_negotiation.cc:299] Negotiation successful
0811 22:28:34.075888 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260811 22:28:34.076282 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:34.062509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:33721 (local address 127.0.0.1:48628)
0811 22:28:34.062700 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:34.062722 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 22:28:34.062828 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:28:34.063141 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:28:34.063146 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:28:34.063698 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:28:34.063923 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:34.063937 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:34.064963 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:34.064969 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:34.066462 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:34.066478 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:34.067192 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:34.067198 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:34.067419 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:34.068003 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:28:34.068026 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:28:34.070796 (+ 2770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:28:34.073772 (+ 2976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:28:34.073778 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:28:34.073782 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:28:34.074277 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:28:34.074626 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:28:34.074631 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:28:34.074635 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:28:34.074800 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:28:34.075308 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:28:34.075314 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:28:34.075475 (+ 161us) client_negotiation.cc:769] Sending connection context
0811 22:28:34.075642 (+ 167us) client_negotiation.cc:240] Negotiation successful
0811 22:28:34.075917 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260811 22:28:34.081157 23011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 601062d9e6af4182b94b5094343e95df: Bootstrap starting.
I20260811 22:28:34.084233 23011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 601062d9e6af4182b94b5094343e95df: Neither blocks nor log segments found. Creating new log.
I20260811 22:28:34.085211 23011 log.cc:824] T 00000000000000000000000000000000 P 601062d9e6af4182b94b5094343e95df: Log is configured to *not* fsync() on all Append() calls
I20260811 22:28:34.087980 23011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 601062d9e6af4182b94b5094343e95df: No bootstrap required, opened a new log
I20260811 22:28:34.095784 23011 raft_consensus.cc:348] T 00000000000000000000000000000000 P 601062d9e6af4182b94b5094343e95df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601062d9e6af4182b94b5094343e95df" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 39669 } } peers { permanent_uuid: "600774ae65ac4617af9542f8549ca7b3" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 39349 } } peers { permanent_uuid: "32d41cb9920d4283900dd689c05850e0" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 33721 } }
I20260811 22:28:34.096064 23011 raft_consensus.cc:374] T 00000000000000000000000000000000 P 601062d9e6af4182b94b5094343e95df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:28:34.096195 23011 raft_consensus.cc:729] T 00000000000000000000000000000000 P 601062d9e6af4182b94b5094343e95df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 601062d9e6af4182b94b5094343e95df, State: Initialized, Role: FOLLOWER
I20260811 22:28:34.096719 23011 consensus_queue.cc:260] T 00000000000000000000000000000000 P 601062d9e6af4182b94b5094343e95df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601062d9e6af4182b94b5094343e95df" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 39669 } } peers { permanent_uuid: "600774ae65ac4617af9542f8549ca7b3" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 39349 } } peers { permanent_uuid: "32d41cb9920d4283900dd689c05850e0" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 33721 } }
I20260811 22:28:34.097623 23338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 601062d9e6af4182b94b5094343e95df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601062d9e6af4182b94b5094343e95df" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 39669 } } peers { permanent_uuid: "600774ae65ac4617af9542f8549ca7b3" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 39349 } } peers { permanent_uuid: "32d41cb9920d4283900dd689c05850e0" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 33721 } } }
I20260811 22:28:34.098016 23338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 601062d9e6af4182b94b5094343e95df [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:28:34.098726 23011 sys_catalog.cc:564] T 00000000000000000000000000000000 P 601062d9e6af4182b94b5094343e95df [sys.catalog]: configured and running, proceeding with master startup.
W20260811 22:28:34.107842 23359 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 601062d9e6af4182b94b5094343e95df: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 22:28:34.108026 23359 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 22:28:34.109627 22893 master_runner.cc:186] Error getting master registration for 127.29.106.61:39349: OK, instance_id { permanent_uuid: "600774ae65ac4617af9542f8549ca7b3" instance_seqno: 1786487313423865 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260811 22:28:34.120229 22893 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260811 22:28:34.142539 23158 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.106.60" port: 33721 } attempt: 5
Aug 11 22:28:34 dist-test-slave-gjrz krb5kdc[22876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487313, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.61@KRBTEST.COM for kudu/127.29.106.60@KRBTEST.COM
I20260811 22:28:34.160501 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:34.143184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:33721 (local address 127.0.0.1:48636)
0811 22:28:34.143503 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:34.143524 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 22:28:34.143635 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:28:34.144001 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:28:34.144007 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:28:34.144585 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:28:34.144808 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:34.144822 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:34.146315 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:34.146321 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:34.147776 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:34.147792 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:34.148828 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:34.148834 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:34.149075 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:34.150570 (+ 1495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:28:34.150593 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:28:34.153437 (+ 2844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:28:34.157647 (+ 4210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:28:34.157652 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:28:34.157656 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:28:34.158200 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:28:34.158632 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:28:34.158637 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:28:34.158641 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:28:34.158819 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:28:34.159486 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:28:34.159491 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:28:34.159662 (+ 171us) client_negotiation.cc:769] Sending connection context
0811 22:28:34.159813 (+ 151us) client_negotiation.cc:240] Negotiation successful
0811 22:28:34.160120 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188}
I20260811 22:28:34.160611 23293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:34.143412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48636 (local address 127.29.106.60:33721)
0811 22:28:34.143601 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:34.143607 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:34.143629 (+ 22us) server_negotiation.cc:407] Connection header received
0811 22:28:34.143758 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:34.143763 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:34.143841 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:34.143953 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:28:34.144979 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:34.146143 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:34.147937 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:34.148649 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:34.148782 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:34.153661 (+ 4879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:28:34.153668 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:28:34.153673 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:28:34.153714 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:28:34.157484 (+ 3770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:34.158341 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:34.158347 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:34.158351 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:34.158518 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:34.158940 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:34.158945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:34.158948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:34.159342 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:28:34.159523 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:34.159968 (+ 445us) server_negotiation.cc:299] Negotiation successful
0811 22:28:34.160227 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260811 22:28:34.164853 23200 tablet.cc:2369] T 00000000000000000000000000000000 P 32d41cb9920d4283900dd689c05850e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:28:34.168737 23158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 600774ae65ac4617af9542f8549ca7b3: Bootstrap starting.
W20260811 22:28:34.169449 22931 tablet.cc:2369] T 00000000000000000000000000000000 P 601062d9e6af4182b94b5094343e95df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:28:34.172837 23158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 600774ae65ac4617af9542f8549ca7b3: Neither blocks nor log segments found. Creating new log.
I20260811 22:28:34.174113 23158 log.cc:824] T 00000000000000000000000000000000 P 600774ae65ac4617af9542f8549ca7b3: Log is configured to *not* fsync() on all Append() calls
I20260811 22:28:34.177254 23158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 600774ae65ac4617af9542f8549ca7b3: No bootstrap required, opened a new log
I20260811 22:28:34.189895 23158 raft_consensus.cc:348] T 00000000000000000000000000000000 P 600774ae65ac4617af9542f8549ca7b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601062d9e6af4182b94b5094343e95df" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 39669 } } peers { permanent_uuid: "600774ae65ac4617af9542f8549ca7b3" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 39349 } } peers { permanent_uuid: "32d41cb9920d4283900dd689c05850e0" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 33721 } }
I20260811 22:28:34.190306 23158 raft_consensus.cc:374] T 00000000000000000000000000000000 P 600774ae65ac4617af9542f8549ca7b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:28:34.190433 23158 raft_consensus.cc:729] T 00000000000000000000000000000000 P 600774ae65ac4617af9542f8549ca7b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 600774ae65ac4617af9542f8549ca7b3, State: Initialized, Role: FOLLOWER
I20260811 22:28:34.191066 23158 consensus_queue.cc:260] T 00000000000000000000000000000000 P 600774ae65ac4617af9542f8549ca7b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601062d9e6af4182b94b5094343e95df" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 39669 } } peers { permanent_uuid: "600774ae65ac4617af9542f8549ca7b3" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 39349 } } peers { permanent_uuid: "32d41cb9920d4283900dd689c05850e0" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 33721 } }
I20260811 22:28:34.192130 23362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 600774ae65ac4617af9542f8549ca7b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601062d9e6af4182b94b5094343e95df" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 39669 } } peers { permanent_uuid: "600774ae65ac4617af9542f8549ca7b3" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 39349 } } peers { permanent_uuid: "32d41cb9920d4283900dd689c05850e0" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 33721 } } }
I20260811 22:28:34.192590 23362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 600774ae65ac4617af9542f8549ca7b3 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:28:34.193367 23158 sys_catalog.cc:564] T 00000000000000000000000000000000 P 600774ae65ac4617af9542f8549ca7b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:28:34.208374 23024 master_runner.cc:263] No leader master found from master 600774ae65ac4617af9542f8549ca7b3
I20260811 22:28:34.208528 23024 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260811 22:28:34.213709 23382 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 600774ae65ac4617af9542f8549ca7b3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 22:28:34.213904 23382 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 22:28:34.337927 23308 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32d41cb9920d4283900dd689c05850e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:28:34.338281 23308 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32d41cb9920d4283900dd689c05850e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601062d9e6af4182b94b5094343e95df" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 39669 } } peers { permanent_uuid: "600774ae65ac4617af9542f8549ca7b3" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 39349 } } peers { permanent_uuid: "32d41cb9920d4283900dd689c05850e0" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 33721 } }
I20260811 22:28:34.340366 23308 leader_election.cc:288] T 00000000000000000000000000000000 P 32d41cb9920d4283900dd689c05850e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 601062d9e6af4182b94b5094343e95df (127.29.106.62:39669), 600774ae65ac4617af9542f8549ca7b3 (127.29.106.61:39349)
I20260811 22:28:34.341610 23114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "32d41cb9920d4283900dd689c05850e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "600774ae65ac4617af9542f8549ca7b3" is_pre_election: true
I20260811 22:28:34.342260 23114 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 600774ae65ac4617af9542f8549ca7b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32d41cb9920d4283900dd689c05850e0 in term 0.
I20260811 22:28:34.343359 23194 leader_election.cc:302] T 00000000000000000000000000000000 P 32d41cb9920d4283900dd689c05850e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32d41cb9920d4283900dd689c05850e0, 600774ae65ac4617af9542f8549ca7b3; no voters:
I20260811 22:28:34.344736 22966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "32d41cb9920d4283900dd689c05850e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "601062d9e6af4182b94b5094343e95df" is_pre_election: true
I20260811 22:28:34.345396 22966 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 601062d9e6af4182b94b5094343e95df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32d41cb9920d4283900dd689c05850e0 in term 0.
I20260811 22:28:34.346395 23308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32d41cb9920d4283900dd689c05850e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:28:34.346577 23308 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32d41cb9920d4283900dd689c05850e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:28:34.346681 23308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32d41cb9920d4283900dd689c05850e0 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:28:34.350220 23308 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32d41cb9920d4283900dd689c05850e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601062d9e6af4182b94b5094343e95df" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 39669 } } peers { permanent_uuid: "600774ae65ac4617af9542f8549ca7b3" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 39349 } } peers { permanent_uuid: "32d41cb9920d4283900dd689c05850e0" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 33721 } }
I20260811 22:28:34.355985 22966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "32d41cb9920d4283900dd689c05850e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "601062d9e6af4182b94b5094343e95df"
I20260811 22:28:34.356211 22966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 601062d9e6af4182b94b5094343e95df [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:28:34.359737 22966 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 601062d9e6af4182b94b5094343e95df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32d41cb9920d4283900dd689c05850e0 in term 1.
I20260811 22:28:34.360246 23195 leader_election.cc:302] T 00000000000000000000000000000000 P 32d41cb9920d4283900dd689c05850e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32d41cb9920d4283900dd689c05850e0, 601062d9e6af4182b94b5094343e95df; no voters:
I20260811 22:28:34.362475 23114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "32d41cb9920d4283900dd689c05850e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "600774ae65ac4617af9542f8549ca7b3"
I20260811 22:28:34.362716 23114 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 600774ae65ac4617af9542f8549ca7b3 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:28:34.366593 23114 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 600774ae65ac4617af9542f8549ca7b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32d41cb9920d4283900dd689c05850e0 in term 1.
I20260811 22:28:34.367405 23308 leader_election.cc:288] T 00000000000000000000000000000000 P 32d41cb9920d4283900dd689c05850e0 [CANDIDATE]: Term 1 election: Requested vote from peers 601062d9e6af4182b94b5094343e95df (127.29.106.62:39669), 600774ae65ac4617af9542f8549ca7b3 (127.29.106.61:39349)
I20260811 22:28:34.367811 23308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32d41cb9920d4283900dd689c05850e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:28:34.368212 23308 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32d41cb9920d4283900dd689c05850e0 [term 1 LEADER]: Becoming Leader. State: Replica: 32d41cb9920d4283900dd689c05850e0, State: Running, Role: LEADER
I20260811 22:28:34.369479 23308 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32d41cb9920d4283900dd689c05850e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601062d9e6af4182b94b5094343e95df" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 39669 } } peers { permanent_uuid: "600774ae65ac4617af9542f8549ca7b3" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 39349 } } peers { permanent_uuid: "32d41cb9920d4283900dd689c05850e0" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 33721 } }
I20260811 22:28:34.384864 23384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32d41cb9920d4283900dd689c05850e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32d41cb9920d4283900dd689c05850e0. Latest consensus state: current_term: 1 leader_uuid: "32d41cb9920d4283900dd689c05850e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601062d9e6af4182b94b5094343e95df" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 39669 } } peers { permanent_uuid: "600774ae65ac4617af9542f8549ca7b3" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 39349 } } peers { permanent_uuid: "32d41cb9920d4283900dd689c05850e0" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 33721 } } }
I20260811 22:28:34.385345 23384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32d41cb9920d4283900dd689c05850e0 [sys.catalog]: This master's current role is: LEADER
I20260811 22:28:34.392141 23388 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:28:34.394548 23388 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:28:34.409725 23114 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 600774ae65ac4617af9542f8549ca7b3 [term 1 FOLLOWER]: Refusing update from remote peer 32d41cb9920d4283900dd689c05850e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:28:34.410818 23384 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 32d41cb9920d4283900dd689c05850e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "600774ae65ac4617af9542f8549ca7b3" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 39349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260811 22:28:34.414194 23072 tablet.cc:2369] T 00000000000000000000000000000000 P 600774ae65ac4617af9542f8549ca7b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:28:34.418433 22966 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 601062d9e6af4182b94b5094343e95df [term 1 FOLLOWER]: Refusing update from remote peer 32d41cb9920d4283900dd689c05850e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:28:34.419248 23384 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 32d41cb9920d4283900dd689c05850e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "601062d9e6af4182b94b5094343e95df" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 39669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:28:34.449023 23338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 601062d9e6af4182b94b5094343e95df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32d41cb9920d4283900dd689c05850e0. Latest consensus state: current_term: 1 leader_uuid: "32d41cb9920d4283900dd689c05850e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601062d9e6af4182b94b5094343e95df" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 39669 } } peers { permanent_uuid: "600774ae65ac4617af9542f8549ca7b3" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 39349 } } peers { permanent_uuid: "32d41cb9920d4283900dd689c05850e0" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 33721 } } }
I20260811 22:28:34.449375 23338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 601062d9e6af4182b94b5094343e95df [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:28:34.459685 23308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32d41cb9920d4283900dd689c05850e0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "32d41cb9920d4283900dd689c05850e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601062d9e6af4182b94b5094343e95df" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 39669 } } peers { permanent_uuid: "600774ae65ac4617af9542f8549ca7b3" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 39349 } } peers { permanent_uuid: "32d41cb9920d4283900dd689c05850e0" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 33721 } } }
I20260811 22:28:34.460039 23308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32d41cb9920d4283900dd689c05850e0 [sys.catalog]: This master's current role is: LEADER
I20260811 22:28:34.460359 23389 mvcc.cc:205] Tried to move back new op lower bound from 7317452039812317184 to 7317452039669080064. Current Snapshot: MvccSnapshot[applied={T|T < 7317452039812317184}]
I20260811 22:28:34.462500 23338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 601062d9e6af4182b94b5094343e95df [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "32d41cb9920d4283900dd689c05850e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601062d9e6af4182b94b5094343e95df" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 39669 } } peers { permanent_uuid: "600774ae65ac4617af9542f8549ca7b3" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 39349 } } peers { permanent_uuid: "32d41cb9920d4283900dd689c05850e0" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 33721 } } }
I20260811 22:28:34.462759 23338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 601062d9e6af4182b94b5094343e95df [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:28:34.464684 23388 catalog_manager.cc:1347] Generated new cluster ID: 2fa923e9408d471996c40b4823b4586f
I20260811 22:28:34.464793 23388 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:28:34.480520 23362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 600774ae65ac4617af9542f8549ca7b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32d41cb9920d4283900dd689c05850e0. Latest consensus state: current_term: 1 leader_uuid: "32d41cb9920d4283900dd689c05850e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601062d9e6af4182b94b5094343e95df" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 39669 } } peers { permanent_uuid: "600774ae65ac4617af9542f8549ca7b3" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 39349 } } peers { permanent_uuid: "32d41cb9920d4283900dd689c05850e0" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 33721 } } }
I20260811 22:28:34.480851 23362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 600774ae65ac4617af9542f8549ca7b3 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:28:34.484288 23384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32d41cb9920d4283900dd689c05850e0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "32d41cb9920d4283900dd689c05850e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601062d9e6af4182b94b5094343e95df" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 39669 } } peers { permanent_uuid: "600774ae65ac4617af9542f8549ca7b3" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 39349 } } peers { permanent_uuid: "32d41cb9920d4283900dd689c05850e0" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 33721 } } }
I20260811 22:28:34.484536 23384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32d41cb9920d4283900dd689c05850e0 [sys.catalog]: This master's current role is: LEADER
I20260811 22:28:34.486327 23390 mvcc.cc:205] Tried to move back new op lower bound from 7317452039812317184 to 7317452039669080064. Current Snapshot: MvccSnapshot[applied={T|T < 7317452039812317184}]
I20260811 22:28:34.488677 23388 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:28:34.490535 23388 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:28:34.499382 23362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 600774ae65ac4617af9542f8549ca7b3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "32d41cb9920d4283900dd689c05850e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "601062d9e6af4182b94b5094343e95df" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 39669 } } peers { permanent_uuid: "600774ae65ac4617af9542f8549ca7b3" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 39349 } } peers { permanent_uuid: "32d41cb9920d4283900dd689c05850e0" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 33721 } } }
I20260811 22:28:34.499603 23362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 600774ae65ac4617af9542f8549ca7b3 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:28:34.507534 23388 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32d41cb9920d4283900dd689c05850e0: Generated new TSK 0
I20260811 22:28:34.508507 23388 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:28:35.110011 23359 catalog_manager.cc:1259] Loaded cluster ID: 2fa923e9408d471996c40b4823b4586f
I20260811 22:28:35.110108 23359 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 601062d9e6af4182b94b5094343e95df: loading cluster ID for follower catalog manager: success
I20260811 22:28:35.112733 23359 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 601062d9e6af4182b94b5094343e95df: acquiring CA information for follower catalog manager: success
I20260811 22:28:35.114559 23359 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 601062d9e6af4182b94b5094343e95df: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 22:28:35.215981 23382 catalog_manager.cc:1259] Loaded cluster ID: 2fa923e9408d471996c40b4823b4586f
I20260811 22:28:35.216096 23382 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 600774ae65ac4617af9542f8549ca7b3: loading cluster ID for follower catalog manager: success
I20260811 22:28:35.218505 23382 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 600774ae65ac4617af9542f8549ca7b3: acquiring CA information for follower catalog manager: success
I20260811 22:28:35.220270 23382 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 600774ae65ac4617af9542f8549ca7b3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 22:28:35.592705 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 22893
I20260811 22:28:35.594020 23005 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:28:35.702795 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 22893
I20260811 22:28:35.732731 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 23024
I20260811 22:28:35.733803 23143 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:28:35.838478 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 23024
I20260811 22:28:35.868276 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 23173
I20260811 22:28:35.869311 23283 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:28:35.977545 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 23173
2026-08-11T22:28:36Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4954 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786487105757681-30120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:28:36 dist-test-slave-gjrz krb5kdc[23417](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:28:36 dist-test-slave-gjrz krb5kdc[23417](info): set up 2 sockets
Aug 11 22:28:36 dist-test-slave-gjrz krb5kdc[23417](info): commencing operation
krb5kdc: starting...
W20260811 22:28:37.252755 30120 mini_kdc.cc:121] Time spent starting KDC: real 1.228s user 0.001s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Aug 11 22:28:37 dist-test-slave-gjrz krb5kdc[23417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:28:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:28:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.106.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
I20260811 22:28:37.499178 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:38987
--webserver_interface=127.29.106.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:45585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:38987
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:28:37.822346 23440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:28:37.822623 23440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:28:37.822753 23440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:28:37.829766 23440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:28:37.829872 23440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:28:37.829948 23440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:28:37.830040 23440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:28:37.830096 23440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:28:37.839504 23440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:45585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:38987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:38987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.106.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:28:37.840651 23440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:28:37.841845 23440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:28:37.852771 23448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:28:37.852834 23450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:28:37.852813 23454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:28:37.853670 23440 server_base.cc:1034] running on GCE node
I20260811 22:28:37.854373 23440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:28:37.855671 23440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:28:37.856870 23440 hybrid_clock.cc:648] HybridClock initialized: now 1786487317856822 us; error 34 us; skew 500 ppm
I20260811 22:28:37.857192 23440 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:28:37 dist-test-slave-gjrz krb5kdc[23417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487317, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:28:37.860265 23440 init.cc:377] Logged in from keytab as kudu/127.29.106.62@KRBTEST.COM (short username kudu)
I20260811 22:28:37.861589 23440 webserver.cc:460] Webserver started at http://127.29.106.62:38451/ using document root <none> and password file <none>
I20260811 22:28:37.862186 23440 fs_manager.cc:359] Metadata directory not provided
I20260811 22:28:37.862286 23440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:28:37.862541 23440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:28:37.865309 23440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "9bd6bd78bcd046ab8aa6c0ee6e0b678a"
format_stamp: "Formatted at 2026-08-11 22:28:37 on dist-test-slave-gjrz"
server_key: "1e876cac2fe264620aa8a2b09790a461"
server_key_iv: "6e7d05d7d1e955c9b62385bf2caf3674"
server_key_version: "encryptionkey@0"
I20260811 22:28:37.865978 23440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "9bd6bd78bcd046ab8aa6c0ee6e0b678a"
format_stamp: "Formatted at 2026-08-11 22:28:37 on dist-test-slave-gjrz"
server_key: "1e876cac2fe264620aa8a2b09790a461"
server_key_iv: "6e7d05d7d1e955c9b62385bf2caf3674"
server_key_version: "encryptionkey@0"
I20260811 22:28:37.870798 23440 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 22:28:37.873988 23468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:28:37.875231 23440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260811 22:28:37.875437 23440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "9bd6bd78bcd046ab8aa6c0ee6e0b678a"
format_stamp: "Formatted at 2026-08-11 22:28:37 on dist-test-slave-gjrz"
server_key: "1e876cac2fe264620aa8a2b09790a461"
server_key_iv: "6e7d05d7d1e955c9b62385bf2caf3674"
server_key_version: "encryptionkey@0"
I20260811 22:28:37.875613 23440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:28:37.894394 23440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:28:37.898437 23440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:28:37.898701 23440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:28:37.916301 23440 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.62:38987
I20260811 22:28:37.916322 23556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.62:38987 every 8 connection(s)
I20260811 22:28:37.917579 23440 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:28:37.920006 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 23440
I20260811 22:28:37.920252 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:28:37.921054 30120 external_mini_cluster.cc:1428] Setting key 34ad468605c84e482082889abdba8e4b
I20260811 22:28:37.921809 23557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 11 22:28:37 dist-test-slave-gjrz krb5kdc[23417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:28:37.936105 23557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bd6bd78bcd046ab8aa6c0ee6e0b678a: Bootstrap starting.
I20260811 22:28:37.939091 23557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bd6bd78bcd046ab8aa6c0ee6e0b678a: Neither blocks nor log segments found. Creating new log.
I20260811 22:28:37.940121 23557 log.cc:824] T 00000000000000000000000000000000 P 9bd6bd78bcd046ab8aa6c0ee6e0b678a: Log is configured to *not* fsync() on all Append() calls
I20260811 22:28:37.943362 23557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bd6bd78bcd046ab8aa6c0ee6e0b678a: No bootstrap required, opened a new log
I20260811 22:28:37.943914 23566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:37.922879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48322 (local address 127.29.106.62:38987)
0811 22:28:37.923667 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:37.923699 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:37.923754 (+ 55us) server_negotiation.cc:407] Connection header received
0811 22:28:37.925373 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:37.925395 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:37.925768 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:37.926093 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:28:37.927967 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:37.929309 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:37.931174 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:37.931876 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:37.932060 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:37.936735 (+ 4675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:28:37.936744 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:28:37.936757 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:28:37.936795 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:28:37.939684 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:37.940324 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:37.940332 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:37.940340 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:37.940541 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:37.940942 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:37.940946 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:37.940948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:37.941516 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:28:37.941797 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:37.942212 (+ 415us) server_negotiation.cc:299] Negotiation successful
0811 22:28:37.942628 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":379,"threads_started":1}
I20260811 22:28:37.947846 30120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:28:37.952818 23557 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bd6bd78bcd046ab8aa6c0ee6e0b678a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd6bd78bcd046ab8aa6c0ee6e0b678a" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 38987 } }
I20260811 22:28:37.953118 23557 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9bd6bd78bcd046ab8aa6c0ee6e0b678a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:28:37.953190 23557 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bd6bd78bcd046ab8aa6c0ee6e0b678a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bd6bd78bcd046ab8aa6c0ee6e0b678a, State: Initialized, Role: FOLLOWER
I20260811 22:28:37.953691 23557 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bd6bd78bcd046ab8aa6c0ee6e0b678a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd6bd78bcd046ab8aa6c0ee6e0b678a" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 38987 } }
I20260811 22:28:37.953894 23557 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9bd6bd78bcd046ab8aa6c0ee6e0b678a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:28:37.953994 23557 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bd6bd78bcd046ab8aa6c0ee6e0b678a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:28:37.954118 23557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bd6bd78bcd046ab8aa6c0ee6e0b678a [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:28:37.956151 23557 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bd6bd78bcd046ab8aa6c0ee6e0b678a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd6bd78bcd046ab8aa6c0ee6e0b678a" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 38987 } }
I20260811 22:28:37.956589 23557 leader_election.cc:302] T 00000000000000000000000000000000 P 9bd6bd78bcd046ab8aa6c0ee6e0b678a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bd6bd78bcd046ab8aa6c0ee6e0b678a; no voters:
I20260811 22:28:37.957075 23557 leader_election.cc:288] T 00000000000000000000000000000000 P 9bd6bd78bcd046ab8aa6c0ee6e0b678a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:28:37.957198 23568 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9bd6bd78bcd046ab8aa6c0ee6e0b678a [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:28:37.957926 23568 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9bd6bd78bcd046ab8aa6c0ee6e0b678a [term 1 LEADER]: Becoming Leader. State: Replica: 9bd6bd78bcd046ab8aa6c0ee6e0b678a, State: Running, Role: LEADER
I20260811 22:28:37.958451 23568 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bd6bd78bcd046ab8aa6c0ee6e0b678a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd6bd78bcd046ab8aa6c0ee6e0b678a" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 38987 } }
I20260811 22:28:37.958983 23557 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bd6bd78bcd046ab8aa6c0ee6e0b678a [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:28:37.961035 23572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bd6bd78bcd046ab8aa6c0ee6e0b678a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bd6bd78bcd046ab8aa6c0ee6e0b678a. Latest consensus state: current_term: 1 leader_uuid: "9bd6bd78bcd046ab8aa6c0ee6e0b678a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd6bd78bcd046ab8aa6c0ee6e0b678a" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 38987 } } }
I20260811 22:28:37.961349 23572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bd6bd78bcd046ab8aa6c0ee6e0b678a [sys.catalog]: This master's current role is: LEADER
I20260811 22:28:37.962007 23569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bd6bd78bcd046ab8aa6c0ee6e0b678a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9bd6bd78bcd046ab8aa6c0ee6e0b678a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd6bd78bcd046ab8aa6c0ee6e0b678a" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 38987 } } }
I20260811 22:28:37.962373 23569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bd6bd78bcd046ab8aa6c0ee6e0b678a [sys.catalog]: This master's current role is: LEADER
I20260811 22:28:37.963145 23579 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:28:37.967978 23579 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:28:37.975996 23579 catalog_manager.cc:1347] Generated new cluster ID: 5c357e9144864ca08aa7b42d5013d1c1
I20260811 22:28:37.976099 23579 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:28:38.015204 23579 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:28:38.016487 23579 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:28:38.026671 23579 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9bd6bd78bcd046ab8aa6c0ee6e0b678a: Generated new TSK 0
I20260811 22:28:38.027495 23579 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:28:38.059849 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 23440
I20260811 22:28:38.060948 23550 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:28:38.154699 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 23440
2026-08-11T22:28:38Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2178 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:28:38 dist-test-slave-gjrz krb5kdc[23604](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:28:38 dist-test-slave-gjrz krb5kdc[23604](info): set up 2 sockets
Aug 11 22:28:38 dist-test-slave-gjrz krb5kdc[23604](info): commencing operation
krb5kdc: starting...
W20260811 22:28:39.423113 30120 mini_kdc.cc:121] Time spent starting KDC: real 1.220s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Aug 11 22:28:39 dist-test-slave-gjrz krb5kdc[23604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:28:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:28:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.106.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
I20260811 22:28:39.675052 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:42607
--webserver_interface=127.29.106.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:40907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:42607
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:28:40.000222 23632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:28:40.000486 23632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:28:40.000610 23632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:28:40.007650 23632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:28:40.007753 23632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:28:40.007824 23632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:28:40.007900 23632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:28:40.007959 23632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:28:40.017382 23632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:40907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:42607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:42607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.106.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:28:40.018546 23632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:28:40.019757 23632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:28:40.032090 23646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:28:40.032058 23647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:28:40.032279 23650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:28:40.032651 23632 server_base.cc:1034] running on GCE node
I20260811 22:28:40.033531 23632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:28:40.034914 23632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:28:40.036144 23632 hybrid_clock.cc:648] HybridClock initialized: now 1786487320036088 us; error 35 us; skew 500 ppm
I20260811 22:28:40.036470 23632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:28:40 dist-test-slave-gjrz krb5kdc[23604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487320, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:28:40.043911 23632 init.cc:377] Logged in from keytab as kudu/127.29.106.62@KRBTEST.COM (short username kudu)
I20260811 22:28:40.045315 23632 webserver.cc:460] Webserver started at http://127.29.106.62:43149/ using document root <none> and password file <none>
I20260811 22:28:40.045925 23632 fs_manager.cc:359] Metadata directory not provided
I20260811 22:28:40.046025 23632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:28:40.046325 23632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:28:40.049233 23632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "d5ab3dad17d944cab5b01b0d6eff043d"
format_stamp: "Formatted at 2026-08-11 22:28:40 on dist-test-slave-gjrz"
server_key: "d44d0be8faa4eb8d8d25f9cdf3764516"
server_key_iv: "c7a0490464ba26dc465190ebf040657f"
server_key_version: "encryptionkey@0"
I20260811 22:28:40.049938 23632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "d5ab3dad17d944cab5b01b0d6eff043d"
format_stamp: "Formatted at 2026-08-11 22:28:40 on dist-test-slave-gjrz"
server_key: "d44d0be8faa4eb8d8d25f9cdf3764516"
server_key_iv: "c7a0490464ba26dc465190ebf040657f"
server_key_version: "encryptionkey@0"
I20260811 22:28:40.054828 23632 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260811 22:28:40.058281 23671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:28:40.059569 23632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:28:40.059713 23632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "d5ab3dad17d944cab5b01b0d6eff043d"
format_stamp: "Formatted at 2026-08-11 22:28:40 on dist-test-slave-gjrz"
server_key: "d44d0be8faa4eb8d8d25f9cdf3764516"
server_key_iv: "c7a0490464ba26dc465190ebf040657f"
server_key_version: "encryptionkey@0"
I20260811 22:28:40.059876 23632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:28:40.071393 23632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:28:40.075344 23632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:28:40.075575 23632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:28:40.092729 23632 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.62:42607
I20260811 22:28:40.092760 23756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.62:42607 every 8 connection(s)
I20260811 22:28:40.093984 23632 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:28:40.097383 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 23632
I20260811 22:28:40.097625 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:28:40.097617 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:28:40.098501 30120 external_mini_cluster.cc:1428] Setting key fe6721c2d08ec1a7a70fd3e7d95c6f3c
I20260811 22:28:40.110896 23759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d: Bootstrap starting.
Aug 11 22:28:40 dist-test-slave-gjrz krb5kdc[23604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:28:40.114559 23759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d: Neither blocks nor log segments found. Creating new log.
I20260811 22:28:40.115763 23759 log.cc:824] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d: Log is configured to *not* fsync() on all Append() calls
I20260811 22:28:40.118881 23759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d: No bootstrap required, opened a new log
I20260811 22:28:40.122135 23768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:40.100543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42574 (local address 127.29.106.62:42607)
0811 22:28:40.101333 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:40.101348 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:40.101396 (+ 48us) server_negotiation.cc:407] Connection header received
0811 22:28:40.103631 (+ 2235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:40.103652 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:40.104013 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:40.104332 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:28:40.106106 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:40.107450 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:40.109259 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:40.109885 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:40.110010 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:40.114669 (+ 4659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:28:40.114681 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:28:40.114696 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:28:40.114731 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:28:40.118098 (+ 3367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:40.118855 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:40.118861 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:40.118871 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:40.119107 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:40.119580 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:40.119583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:40.119585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:40.120086 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:28:40.120314 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:40.120780 (+ 466us) server_negotiation.cc:299] Negotiation successful
0811 22:28:40.121134 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":337,"threads_started":1}
I20260811 22:28:40.126856 23759 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ab3dad17d944cab5b01b0d6eff043d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 42607 } }
I20260811 22:28:40.127184 23759 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:28:40.127290 23759 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5ab3dad17d944cab5b01b0d6eff043d, State: Initialized, Role: FOLLOWER
I20260811 22:28:40.127893 23759 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ab3dad17d944cab5b01b0d6eff043d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 42607 } }
I20260811 22:28:40.128113 23759 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:28:40.128211 23759 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:28:40.128335 23759 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:28:40.130301 23759 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ab3dad17d944cab5b01b0d6eff043d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 42607 } }
I20260811 22:28:40.130812 23759 leader_election.cc:302] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5ab3dad17d944cab5b01b0d6eff043d; no voters:
I20260811 22:28:40.131429 23759 leader_election.cc:288] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:28:40.131524 23777 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:28:40.132205 23777 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [term 1 LEADER]: Becoming Leader. State: Replica: d5ab3dad17d944cab5b01b0d6eff043d, State: Running, Role: LEADER
I20260811 22:28:40.132737 23777 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ab3dad17d944cab5b01b0d6eff043d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 42607 } }
I20260811 22:28:40.133332 23759 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:28:40.135243 23784 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5ab3dad17d944cab5b01b0d6eff043d. Latest consensus state: current_term: 1 leader_uuid: "d5ab3dad17d944cab5b01b0d6eff043d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ab3dad17d944cab5b01b0d6eff043d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 42607 } } }
I20260811 22:28:40.135676 23784 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [sys.catalog]: This master's current role is: LEADER
I20260811 22:28:40.139459 23783 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5ab3dad17d944cab5b01b0d6eff043d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ab3dad17d944cab5b01b0d6eff043d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 42607 } } }
I20260811 22:28:40.139853 23783 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [sys.catalog]: This master's current role is: LEADER
I20260811 22:28:40.141072 23794 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:28:40.146303 23794 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:28:40.154510 23794 catalog_manager.cc:1347] Generated new cluster ID: a4952745047642089b4a0fcfdfcfd184
I20260811 22:28:40.154608 23794 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:28:40.167799 23794 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:28:40.169534 23794 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:28:40.177598 23794 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d: Generated new TSK 0
I20260811 22:28:40.178510 23794 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.106.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab.
Entry for principal kudu/127.29.106.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab.
I20260811 22:28:40.240811 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.1:0
--local_ip_for_outbound_sockets=127.29.106.1
--webserver_interface=127.29.106.1
--webserver_port=0
--tserver_master_addrs=127.29.106.62:42607
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab
--principal=kudu/127.29.106.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:40907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:28:40.566035 23820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:28:40.566295 23820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:28:40.566430 23820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:28:40.573344 23820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:28:40.573454 23820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:28:40.573607 23820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.1
I20260811 22:28:40.582784 23820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:40907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab
--principal=kudu/127.29.106.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.106.1
--webserver_port=0
--tserver_master_addrs=127.29.106.62:42607
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.1
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:28:40.583891 23820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:28:40.585115 23820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:28:40.597240 23838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:28:40.597285 23831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:28:40.597323 23830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:28:40.598134 23820 server_base.cc:1034] running on GCE node
I20260811 22:28:40.598671 23820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:28:40.599555 23820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:28:40.600749 23820 hybrid_clock.cc:648] HybridClock initialized: now 1786487320600701 us; error 42 us; skew 500 ppm
I20260811 22:28:40.601024 23820 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:28:40 dist-test-slave-gjrz krb5kdc[23604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487320, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:28:40.604063 23820 init.cc:377] Logged in from keytab as kudu/127.29.106.1@KRBTEST.COM (short username kudu)
I20260811 22:28:40.605392 23820 webserver.cc:460] Webserver started at http://127.29.106.1:40377/ using document root <none> and password file <none>
I20260811 22:28:40.606032 23820 fs_manager.cc:359] Metadata directory not provided
I20260811 22:28:40.606150 23820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:28:40.606428 23820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:28:40.609179 23820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-0/data/instance:
uuid: "7bf0fa1e88d04e0397be79b9099418d5"
format_stamp: "Formatted at 2026-08-11 22:28:40 on dist-test-slave-gjrz"
server_key: "56282eed7170f61813e443af894ab68d"
server_key_iv: "f0ff4839e78c33dba7882b0666c8e859"
server_key_version: "encryptionkey@0"
I20260811 22:28:40.609937 23820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance:
uuid: "7bf0fa1e88d04e0397be79b9099418d5"
format_stamp: "Formatted at 2026-08-11 22:28:40 on dist-test-slave-gjrz"
server_key: "56282eed7170f61813e443af894ab68d"
server_key_iv: "f0ff4839e78c33dba7882b0666c8e859"
server_key_version: "encryptionkey@0"
I20260811 22:28:40.614657 23820 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260811 22:28:40.617993 23851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:28:40.619181 23820 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 22:28:40.619372 23820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-0/wal
uuid: "7bf0fa1e88d04e0397be79b9099418d5"
format_stamp: "Formatted at 2026-08-11 22:28:40 on dist-test-slave-gjrz"
server_key: "56282eed7170f61813e443af894ab68d"
server_key_iv: "f0ff4839e78c33dba7882b0666c8e859"
server_key_version: "encryptionkey@0"
I20260811 22:28:40.619556 23820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:28:40.641052 23820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:28:40.644632 23820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:28:40.644879 23820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:28:40.645720 23820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:28:40.646961 23820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:28:40.647047 23820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:28:40.647166 23820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:28:40.647219 23820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:28:40.676571 23820 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.1:46373
I20260811 22:28:40.676627 24058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.1:46373 every 8 connection(s)
I20260811 22:28:40.677763 23820 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
I20260811 22:28:40.683879 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 23820
I20260811 22:28:40.684065 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance
I20260811 22:28:40.684667 30120 external_mini_cluster.cc:1428] Setting key 7c0204c75b5adc3239ce6985a3609ca7
Aug 11 22:28:40 dist-test-slave-gjrz krb5kdc[23604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487320, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.1@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:28:40.696295 24065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:40.680069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:54343 (local address 127.29.106.62:42607)
0811 22:28:40.680480 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:40.680484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:40.681387 (+ 903us) server_negotiation.cc:407] Connection header received
0811 22:28:40.683196 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:40.683201 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:40.683329 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:40.683425 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:28:40.684932 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:40.685762 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:40.687431 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:40.687838 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:40.687947 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:40.691078 (+ 3131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:28:40.691083 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:28:40.691087 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:28:40.691120 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:28:40.693465 (+ 2345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:40.694121 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:40.694127 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:40.694130 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:40.694250 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:40.694647 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:40.694653 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:40.694655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:40.694916 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:28:40.695051 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:40.695737 (+ 686us) server_negotiation.cc:299] Negotiation successful
0811 22:28:40.695961 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":211,"threads_started":1}
I20260811 22:28:40.697252 24066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:40.680568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.1:54343)
0811 22:28:40.681240 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:40.681282 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0811 22:28:40.682925 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:28:40.683678 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:28:40.683686 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:28:40.684220 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:28:40.684752 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:40.684776 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:40.685907 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:40.685911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:40.687286 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:40.687304 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:40.688047 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:40.688065 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:40.688281 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:40.688859 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:28:40.688880 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:28:40.690878 (+ 1998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:28:40.693595 (+ 2717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:28:40.693604 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:28:40.693618 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:28:40.694009 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:28:40.694351 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:28:40.694356 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:28:40.694360 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:28:40.694538 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:28:40.695041 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:28:40.695049 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:28:40.695383 (+ 334us) client_negotiation.cc:769] Sending connection context
0811 22:28:40.695617 (+ 234us) client_negotiation.cc:240] Negotiation successful
0811 22:28:40.696089 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":250,"threads_started":1}
I20260811 22:28:40.698590 24060 heartbeater.cc:347] Connected to a master server at 127.29.106.62:42607
I20260811 22:28:40.698851 24060 heartbeater.cc:464] Registering TS with master...
I20260811 22:28:40.699545 24060 heartbeater.cc:511] Master 127.29.106.62:42607 requested a full tablet report, sending...
I20260811 22:28:40.701509 23697 ts_manager.cc:194] Registered new tserver with Master: 7bf0fa1e88d04e0397be79b9099418d5 (127.29.106.1:46373)
I20260811 22:28:40.703023 23697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.1@KRBTEST.COM'} at 127.29.106.1:54343
WARNING: no policy specified for kudu/127.29.106.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab.
Entry for principal kudu/127.29.106.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab.
I20260811 22:28:40.738294 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.2:0
--local_ip_for_outbound_sockets=127.29.106.2
--webserver_interface=127.29.106.2
--webserver_port=0
--tserver_master_addrs=127.29.106.62:42607
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab
--principal=kudu/127.29.106.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:40907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:28:41.057632 24072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:28:41.057886 24072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:28:41.057977 24072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:28:41.065357 24072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:28:41.065467 24072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:28:41.065590 24072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.2
I20260811 22:28:41.075237 24072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:40907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab
--principal=kudu/127.29.106.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.106.2
--webserver_port=0
--tserver_master_addrs=127.29.106.62:42607
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.2
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:28:41.076419 24072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:28:41.077736 24072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:28:41.090215 24081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:28:41.090168 24089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:28:41.090241 24082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:28:41.091017 24072 server_base.cc:1034] running on GCE node
I20260811 22:28:41.091578 24072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:28:41.092494 24072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:28:41.093706 24072 hybrid_clock.cc:648] HybridClock initialized: now 1786487321093644 us; error 54 us; skew 500 ppm
I20260811 22:28:41.094017 24072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:28:41 dist-test-slave-gjrz krb5kdc[23604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487321, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:28:41.097038 24072 init.cc:377] Logged in from keytab as kudu/127.29.106.2@KRBTEST.COM (short username kudu)
I20260811 22:28:41.098402 24072 webserver.cc:460] Webserver started at http://127.29.106.2:40101/ using document root <none> and password file <none>
I20260811 22:28:41.099046 24072 fs_manager.cc:359] Metadata directory not provided
I20260811 22:28:41.099155 24072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:28:41.099481 24072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:28:41.102269 24072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-1/data/instance:
uuid: "bfb4616fe5fb4d5d8ffb2a940d5e2358"
format_stamp: "Formatted at 2026-08-11 22:28:41 on dist-test-slave-gjrz"
server_key: "4f13b02c6438febf776ce8441d9de1b4"
server_key_iv: "5bd54651f209888a484ba02d8f8abbc2"
server_key_version: "encryptionkey@0"
I20260811 22:28:41.103050 24072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance:
uuid: "bfb4616fe5fb4d5d8ffb2a940d5e2358"
format_stamp: "Formatted at 2026-08-11 22:28:41 on dist-test-slave-gjrz"
server_key: "4f13b02c6438febf776ce8441d9de1b4"
server_key_iv: "5bd54651f209888a484ba02d8f8abbc2"
server_key_version: "encryptionkey@0"
I20260811 22:28:41.107754 24072 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260811 22:28:41.110927 24110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:28:41.112145 24072 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:28:41.112285 24072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-1/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-1/wal
uuid: "bfb4616fe5fb4d5d8ffb2a940d5e2358"
format_stamp: "Formatted at 2026-08-11 22:28:41 on dist-test-slave-gjrz"
server_key: "4f13b02c6438febf776ce8441d9de1b4"
server_key_iv: "5bd54651f209888a484ba02d8f8abbc2"
server_key_version: "encryptionkey@0"
I20260811 22:28:41.112459 24072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:28:41.137542 24072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:28:41.141603 24072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:28:41.141860 24072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:28:41.142774 24072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:28:41.144059 24072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:28:41.144135 24072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:28:41.144248 24072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:28:41.144297 24072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:28:41.173324 24072 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.2:44241
I20260811 22:28:41.173368 24318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.2:44241 every 8 connection(s)
I20260811 22:28:41.174587 24072 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
I20260811 22:28:41.178594 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 24072
I20260811 22:28:41.178771 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance
I20260811 22:28:41.179198 30120 external_mini_cluster.cc:1428] Setting key 65399a064e12d4955d46c26e37b7cb9e
Aug 11 22:28:41 dist-test-slave-gjrz krb5kdc[23604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487321, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.2@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:28:41.193986 24065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:41.177086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:38231 (local address 127.29.106.62:42607)
0811 22:28:41.177269 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:41.177273 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:41.178496 (+ 1223us) server_negotiation.cc:407] Connection header received
0811 22:28:41.180478 (+ 1982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:41.180483 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:41.180557 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:41.180672 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:28:41.182112 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:41.182817 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:41.184381 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:41.184807 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:41.184918 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:41.188360 (+ 3442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:28:41.188365 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:28:41.188368 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:28:41.188400 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:28:41.190829 (+ 2429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:41.191653 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:41.191657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:41.191659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:41.191797 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:41.192257 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:41.192261 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:41.192264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:41.192625 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:28:41.192765 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:41.193435 (+ 670us) server_negotiation.cc:299] Negotiation successful
0811 22:28:41.193663 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260811 22:28:41.194664 24324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:41.177489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.2:38231)
0811 22:28:41.178310 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:41.178365 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0811 22:28:41.180229 (+ 1864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:28:41.180838 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:28:41.180845 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:28:41.181397 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:28:41.181947 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:41.181966 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:41.182966 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:41.182970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:41.184237 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:41.184255 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:41.185020 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:41.185028 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:41.185246 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:41.185838 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:28:41.185859 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:28:41.188158 (+ 2299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:28:41.190967 (+ 2809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:28:41.190975 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:28:41.190989 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:28:41.191548 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:28:41.191896 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:28:41.191900 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:28:41.191903 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:28:41.192150 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:28:41.192764 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:28:41.192772 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:28:41.193084 (+ 312us) client_negotiation.cc:769] Sending connection context
0811 22:28:41.193301 (+ 217us) client_negotiation.cc:240] Negotiation successful
0811 22:28:41.193664 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":327,"threads_started":1}
I20260811 22:28:41.196137 24320 heartbeater.cc:347] Connected to a master server at 127.29.106.62:42607
I20260811 22:28:41.196420 24320 heartbeater.cc:464] Registering TS with master...
I20260811 22:28:41.197100 24320 heartbeater.cc:511] Master 127.29.106.62:42607 requested a full tablet report, sending...
I20260811 22:28:41.198508 23697 ts_manager.cc:194] Registered new tserver with Master: bfb4616fe5fb4d5d8ffb2a940d5e2358 (127.29.106.2:44241)
I20260811 22:28:41.199327 23697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.2@KRBTEST.COM'} at 127.29.106.2:38231
WARNING: no policy specified for kudu/127.29.106.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab.
Entry for principal kudu/127.29.106.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab.
I20260811 22:28:41.236150 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.3:0
--local_ip_for_outbound_sockets=127.29.106.3
--webserver_interface=127.29.106.3
--webserver_port=0
--tserver_master_addrs=127.29.106.62:42607
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab
--principal=kudu/127.29.106.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:40907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:28:41.552490 24334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:28:41.552750 24334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:28:41.552889 24334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:28:41.559842 24334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:28:41.559957 24334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:28:41.560106 24334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.3
I20260811 22:28:41.569473 24334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:40907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab
--principal=kudu/127.29.106.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.106.3
--webserver_port=0
--tserver_master_addrs=127.29.106.62:42607
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.3
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:28:41.570616 24334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:28:41.571920 24334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:28:41.584256 24344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:28:41.584296 24346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:28:41.584445 24349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:28:41.585297 24334 server_base.cc:1034] running on GCE node
I20260811 22:28:41.585841 24334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:28:41.586687 24334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:28:41.587883 24334 hybrid_clock.cc:648] HybridClock initialized: now 1786487321587831 us; error 45 us; skew 500 ppm
I20260811 22:28:41.588181 24334 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:28:41 dist-test-slave-gjrz krb5kdc[23604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487321, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:28:41.591150 24334 init.cc:377] Logged in from keytab as kudu/127.29.106.3@KRBTEST.COM (short username kudu)
I20260811 22:28:41.592478 24334 webserver.cc:460] Webserver started at http://127.29.106.3:45753/ using document root <none> and password file <none>
I20260811 22:28:41.593123 24334 fs_manager.cc:359] Metadata directory not provided
I20260811 22:28:41.593242 24334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:28:41.593532 24334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:28:41.596335 24334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-2/data/instance:
uuid: "5470802c25f546c4b8955738575876b8"
format_stamp: "Formatted at 2026-08-11 22:28:41 on dist-test-slave-gjrz"
server_key: "8b6b27dc3bd88b4a88c6ae6fcaa0015a"
server_key_iv: "69684c349d7dd5247f2bddfd3b31e38b"
server_key_version: "encryptionkey@0"
I20260811 22:28:41.597077 24334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance:
uuid: "5470802c25f546c4b8955738575876b8"
format_stamp: "Formatted at 2026-08-11 22:28:41 on dist-test-slave-gjrz"
server_key: "8b6b27dc3bd88b4a88c6ae6fcaa0015a"
server_key_iv: "69684c349d7dd5247f2bddfd3b31e38b"
server_key_version: "encryptionkey@0"
I20260811 22:28:41.601976 24334 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260811 22:28:41.605218 24365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:28:41.606405 24334 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 22:28:41.606554 24334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-2/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-2/wal
uuid: "5470802c25f546c4b8955738575876b8"
format_stamp: "Formatted at 2026-08-11 22:28:41 on dist-test-slave-gjrz"
server_key: "8b6b27dc3bd88b4a88c6ae6fcaa0015a"
server_key_iv: "69684c349d7dd5247f2bddfd3b31e38b"
server_key_version: "encryptionkey@0"
I20260811 22:28:41.606735 24334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:28:41.622290 24334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:28:41.626209 24334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:28:41.626457 24334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:28:41.627331 24334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:28:41.628660 24334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:28:41.628736 24334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:28:41.628852 24334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:28:41.628904 24334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:28:41.658824 24334 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.3:41165
I20260811 22:28:41.658876 24577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.3:41165 every 8 connection(s)
I20260811 22:28:41.660074 24334 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
I20260811 22:28:41.667562 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 24334
I20260811 22:28:41.667750 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance
I20260811 22:28:41.668174 30120 external_mini_cluster.cc:1428] Setting key a1410df611f2a160a2ec8445e08a2b70
Aug 11 22:28:41 dist-test-slave-gjrz krb5kdc[23604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487321, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.3@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:28:41.679363 24065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:41.662447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:46025 (local address 127.29.106.62:42607)
0811 22:28:41.662614 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:41.662619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:41.663774 (+ 1155us) server_negotiation.cc:407] Connection header received
0811 22:28:41.665656 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:41.665662 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:41.665736 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:41.665826 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:28:41.667294 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:41.668150 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:41.670001 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:41.670517 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:41.670618 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:41.673887 (+ 3269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:28:41.673892 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:28:41.673895 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:28:41.673928 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:28:41.676531 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:41.677205 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:41.677210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:41.677214 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:41.677342 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:41.677746 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:41.677750 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:41.677752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:41.678029 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:28:41.678163 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:41.678778 (+ 615us) server_negotiation.cc:299] Negotiation successful
0811 22:28:41.678998 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260811 22:28:41.680243 24581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:41.662831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.3:46025)
0811 22:28:41.663623 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:41.663677 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0811 22:28:41.665354 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:28:41.666066 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:28:41.666074 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:28:41.666540 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:28:41.667100 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:41.667118 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:41.668304 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:41.668308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:41.669851 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:41.669867 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:41.670740 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:41.670748 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:41.671041 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:41.671650 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:28:41.671672 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:28:41.673700 (+ 2028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:28:41.676671 (+ 2971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:28:41.676680 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:28:41.676694 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:28:41.677093 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:28:41.677446 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:28:41.677451 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:28:41.677455 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:28:41.677640 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:28:41.678152 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:28:41.678161 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:28:41.678447 (+ 286us) client_negotiation.cc:769] Sending connection context
0811 22:28:41.678669 (+ 222us) client_negotiation.cc:240] Negotiation successful
0811 22:28:41.679153 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":281,"threads_started":1}
I20260811 22:28:41.681572 24578 heartbeater.cc:347] Connected to a master server at 127.29.106.62:42607
I20260811 22:28:41.681854 24578 heartbeater.cc:464] Registering TS with master...
I20260811 22:28:41.682698 24578 heartbeater.cc:511] Master 127.29.106.62:42607 requested a full tablet report, sending...
I20260811 22:28:41.684158 23697 ts_manager.cc:194] Registered new tserver with Master: 5470802c25f546c4b8955738575876b8 (127.29.106.3:41165)
I20260811 22:28:41.684900 23697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.3@KRBTEST.COM'} at 127.29.106.3:46025
I20260811 22:28:41.694940 30120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 22:28:41.705662 24060 heartbeater.cc:503] Master 127.29.106.62:42607 was elected leader, sending a full tablet report...
I20260811 22:28:41.712684 24065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:41.699222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42580 (local address 127.29.106.62:42607)
0811 22:28:41.699433 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:41.699439 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:41.699752 (+ 313us) server_negotiation.cc:407] Connection header received
0811 22:28:41.699904 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:41.699908 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:41.699974 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:41.700051 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:28:41.701419 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:41.702136 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:41.703446 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:41.703862 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:41.703971 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:41.706674 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:28:41.706680 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:28:41.706684 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:28:41.706718 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:28:41.709769 (+ 3051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:41.710569 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:41.710574 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:41.710578 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:41.710739 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:41.711183 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:41.711188 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:41.711191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:41.711606 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:28:41.711815 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:41.712267 (+ 452us) server_negotiation.cc:299] Negotiation successful
0811 22:28:41.712440 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260811 22:28:41.723793 23697 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42580:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:28:41.730501 23697 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:28:41.752753 24608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:41.742252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57926 (local address 127.29.106.2:44241)
0811 22:28:41.744387 (+ 2135us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:41.744395 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:41.744423 (+ 28us) server_negotiation.cc:407] Connection header received
0811 22:28:41.744519 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:41.744527 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:41.744794 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:41.744985 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:28:41.745981 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:41.747637 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:41.750101 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:41.751590 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:41.751735 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:41.751929 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:41.752257 (+ 328us) server_negotiation.cc:299] Negotiation successful
0811 22:28:41.752431 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1972,"thread_start_us":1808,"threads_started":1}
I20260811 22:28:41.752810 24604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:41.742029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:46373 (local address 127.0.0.1:60608)
0811 22:28:41.744504 (+ 2475us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:41.744517 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:28:41.744645 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:28:41.745565 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:28:41.745585 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:28:41.745608 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:28:41.746104 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:41.746123 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:41.747972 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:41.747977 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:41.750159 (+ 2182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:41.750174 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:41.752091 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:41.752098 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:41.752256 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:41.752288 (+ 32us) client_negotiation.cc:769] Sending connection context
0811 22:28:41.752373 (+ 85us) client_negotiation.cc:240] Negotiation successful
0811 22:28:41.752483 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2387,"thread_start_us":2269,"threads_started":1}
I20260811 22:28:41.753638 24606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:41.742264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60608 (local address 127.29.106.1:46373)
0811 22:28:41.743330 (+ 1066us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:41.743339 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:41.744774 (+ 1435us) server_negotiation.cc:407] Connection header received
0811 22:28:41.744874 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:41.744881 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:41.745104 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:41.745298 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:28:41.746349 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:41.747854 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:41.750348 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:41.751754 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:41.751875 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:41.752067 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:41.753181 (+ 1114us) server_negotiation.cc:299] Negotiation successful
0811 22:28:41.753348 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":881,"thread_start_us":721,"threads_started":1}
I20260811 22:28:41.753782 24603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:41.741819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.2:44241 (local address 127.0.0.1:57926)
0811 22:28:41.743629 (+ 1810us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:41.743659 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0811 22:28:41.743798 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:28:41.745301 (+ 1503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:28:41.745309 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:28:41.745341 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:28:41.745780 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:41.745793 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:41.747808 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:41.747813 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:41.749942 (+ 2129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:41.749956 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:41.751807 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:41.751814 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:41.751996 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:41.752049 (+ 53us) client_negotiation.cc:769] Sending connection context
0811 22:28:41.752194 (+ 145us) client_negotiation.cc:240] Negotiation successful
0811 22:28:41.752371 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1619,"mutex_wait_us":23,"thread_start_us":1504,"threads_started":1}
I20260811 22:28:41.753904 24609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:41.744663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.3:41165 (local address 127.0.0.1:55416)
0811 22:28:41.746361 (+ 1698us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:41.746379 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 22:28:41.746489 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:28:41.747483 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:28:41.747487 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:28:41.747509 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:28:41.747916 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:41.747926 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:41.749871 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:41.749876 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:41.751247 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:41.751283 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:41.753158 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:41.753164 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:41.753321 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:41.753352 (+ 31us) client_negotiation.cc:769] Sending connection context
0811 22:28:41.753436 (+ 84us) client_negotiation.cc:240] Negotiation successful
0811 22:28:41.753573 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1564,"mutex_wait_us":2,"thread_start_us":1427,"threads_started":1}
I20260811 22:28:41.754539 24610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:41.745039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55416 (local address 127.29.106.3:41165)
0811 22:28:41.747036 (+ 1997us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:41.747041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:41.747067 (+ 26us) server_negotiation.cc:407] Connection header received
0811 22:28:41.747140 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:41.747145 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:41.747320 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:41.747482 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:28:41.748087 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:41.749717 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:41.751421 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:41.752682 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:41.752798 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:41.753009 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:41.754122 (+ 1113us) server_negotiation.cc:299] Negotiation successful
0811 22:28:41.754279 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1865,"thread_start_us":1677,"threads_started":1}
I20260811 22:28:41.756992 23932 tablet_service.cc:1467] Processing CreateTablet for tablet d71c0b6bd70c42afa03645c32000f1ba (DEFAULT_TABLE table=test-table [id=acfe615bb1124fe3a4c5f1d528269f47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:28:41.757283 24198 tablet_service.cc:1467] Processing CreateTablet for tablet d71c0b6bd70c42afa03645c32000f1ba (DEFAULT_TABLE table=test-table [id=acfe615bb1124fe3a4c5f1d528269f47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:28:41.757841 24451 tablet_service.cc:1467] Processing CreateTablet for tablet d71c0b6bd70c42afa03645c32000f1ba (DEFAULT_TABLE table=test-table [id=acfe615bb1124fe3a4c5f1d528269f47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:28:41.758406 23932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d71c0b6bd70c42afa03645c32000f1ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:28:41.758633 24198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d71c0b6bd70c42afa03645c32000f1ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:28:41.759181 24451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d71c0b6bd70c42afa03645c32000f1ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:28:41.769074 24611 tablet_bootstrap.cc:492] T d71c0b6bd70c42afa03645c32000f1ba P 7bf0fa1e88d04e0397be79b9099418d5: Bootstrap starting.
I20260811 22:28:41.772639 24611 tablet_bootstrap.cc:654] T d71c0b6bd70c42afa03645c32000f1ba P 7bf0fa1e88d04e0397be79b9099418d5: Neither blocks nor log segments found. Creating new log.
I20260811 22:28:41.772717 24614 tablet_bootstrap.cc:492] T d71c0b6bd70c42afa03645c32000f1ba P 5470802c25f546c4b8955738575876b8: Bootstrap starting.
I20260811 22:28:41.773424 24617 tablet_bootstrap.cc:492] T d71c0b6bd70c42afa03645c32000f1ba P bfb4616fe5fb4d5d8ffb2a940d5e2358: Bootstrap starting.
I20260811 22:28:41.774001 24611 log.cc:824] T d71c0b6bd70c42afa03645c32000f1ba P 7bf0fa1e88d04e0397be79b9099418d5: Log is configured to *not* fsync() on all Append() calls
I20260811 22:28:41.776732 24614 tablet_bootstrap.cc:654] T d71c0b6bd70c42afa03645c32000f1ba P 5470802c25f546c4b8955738575876b8: Neither blocks nor log segments found. Creating new log.
I20260811 22:28:41.776739 24617 tablet_bootstrap.cc:654] T d71c0b6bd70c42afa03645c32000f1ba P bfb4616fe5fb4d5d8ffb2a940d5e2358: Neither blocks nor log segments found. Creating new log.
I20260811 22:28:41.777241 24611 tablet_bootstrap.cc:492] T d71c0b6bd70c42afa03645c32000f1ba P 7bf0fa1e88d04e0397be79b9099418d5: No bootstrap required, opened a new log
I20260811 22:28:41.777505 24611 ts_tablet_manager.cc:1397] T d71c0b6bd70c42afa03645c32000f1ba P 7bf0fa1e88d04e0397be79b9099418d5: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260811 22:28:41.777712 24614 log.cc:824] T d71c0b6bd70c42afa03645c32000f1ba P 5470802c25f546c4b8955738575876b8: Log is configured to *not* fsync() on all Append() calls
I20260811 22:28:41.777918 24617 log.cc:824] T d71c0b6bd70c42afa03645c32000f1ba P bfb4616fe5fb4d5d8ffb2a940d5e2358: Log is configured to *not* fsync() on all Append() calls
I20260811 22:28:41.780392 24614 tablet_bootstrap.cc:492] T d71c0b6bd70c42afa03645c32000f1ba P 5470802c25f546c4b8955738575876b8: No bootstrap required, opened a new log
I20260811 22:28:41.780623 24614 ts_tablet_manager.cc:1397] T d71c0b6bd70c42afa03645c32000f1ba P 5470802c25f546c4b8955738575876b8: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260811 22:28:41.785017 24617 tablet_bootstrap.cc:492] T d71c0b6bd70c42afa03645c32000f1ba P bfb4616fe5fb4d5d8ffb2a940d5e2358: No bootstrap required, opened a new log
I20260811 22:28:41.785264 24617 ts_tablet_manager.cc:1397] T d71c0b6bd70c42afa03645c32000f1ba P bfb4616fe5fb4d5d8ffb2a940d5e2358: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20260811 22:28:41.786275 24611 raft_consensus.cc:348] T d71c0b6bd70c42afa03645c32000f1ba P 7bf0fa1e88d04e0397be79b9099418d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb4616fe5fb4d5d8ffb2a940d5e2358" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44241 } } peers { permanent_uuid: "7bf0fa1e88d04e0397be79b9099418d5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 46373 } } peers { permanent_uuid: "5470802c25f546c4b8955738575876b8" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41165 } }
I20260811 22:28:41.786705 24611 raft_consensus.cc:374] T d71c0b6bd70c42afa03645c32000f1ba P 7bf0fa1e88d04e0397be79b9099418d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:28:41.786800 24611 raft_consensus.cc:729] T d71c0b6bd70c42afa03645c32000f1ba P 7bf0fa1e88d04e0397be79b9099418d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bf0fa1e88d04e0397be79b9099418d5, State: Initialized, Role: FOLLOWER
I20260811 22:28:41.787423 24611 consensus_queue.cc:260] T d71c0b6bd70c42afa03645c32000f1ba P 7bf0fa1e88d04e0397be79b9099418d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb4616fe5fb4d5d8ffb2a940d5e2358" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44241 } } peers { permanent_uuid: "7bf0fa1e88d04e0397be79b9099418d5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 46373 } } peers { permanent_uuid: "5470802c25f546c4b8955738575876b8" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41165 } }
I20260811 22:28:41.787432 24614 raft_consensus.cc:348] T d71c0b6bd70c42afa03645c32000f1ba P 5470802c25f546c4b8955738575876b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb4616fe5fb4d5d8ffb2a940d5e2358" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44241 } } peers { permanent_uuid: "7bf0fa1e88d04e0397be79b9099418d5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 46373 } } peers { permanent_uuid: "5470802c25f546c4b8955738575876b8" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41165 } }
I20260811 22:28:41.787734 24614 raft_consensus.cc:374] T d71c0b6bd70c42afa03645c32000f1ba P 5470802c25f546c4b8955738575876b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:28:41.787838 24614 raft_consensus.cc:729] T d71c0b6bd70c42afa03645c32000f1ba P 5470802c25f546c4b8955738575876b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5470802c25f546c4b8955738575876b8, State: Initialized, Role: FOLLOWER
I20260811 22:28:41.788496 24614 consensus_queue.cc:260] T d71c0b6bd70c42afa03645c32000f1ba P 5470802c25f546c4b8955738575876b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb4616fe5fb4d5d8ffb2a940d5e2358" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44241 } } peers { permanent_uuid: "7bf0fa1e88d04e0397be79b9099418d5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 46373 } } peers { permanent_uuid: "5470802c25f546c4b8955738575876b8" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41165 } }
I20260811 22:28:41.789062 24611 ts_tablet_manager.cc:1428] T d71c0b6bd70c42afa03645c32000f1ba P 7bf0fa1e88d04e0397be79b9099418d5: Time spent starting tablet: real 0.011s user 0.003s sys 0.009s
I20260811 22:28:41.789444 24578 heartbeater.cc:503] Master 127.29.106.62:42607 was elected leader, sending a full tablet report...
I20260811 22:28:41.789974 24614 ts_tablet_manager.cc:1428] T d71c0b6bd70c42afa03645c32000f1ba P 5470802c25f546c4b8955738575876b8: Time spent starting tablet: real 0.009s user 0.003s sys 0.006s
I20260811 22:28:41.796260 24617 raft_consensus.cc:348] T d71c0b6bd70c42afa03645c32000f1ba P bfb4616fe5fb4d5d8ffb2a940d5e2358 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb4616fe5fb4d5d8ffb2a940d5e2358" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44241 } } peers { permanent_uuid: "7bf0fa1e88d04e0397be79b9099418d5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 46373 } } peers { permanent_uuid: "5470802c25f546c4b8955738575876b8" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41165 } }
I20260811 22:28:41.796554 24617 raft_consensus.cc:374] T d71c0b6bd70c42afa03645c32000f1ba P bfb4616fe5fb4d5d8ffb2a940d5e2358 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:28:41.796657 24617 raft_consensus.cc:729] T d71c0b6bd70c42afa03645c32000f1ba P bfb4616fe5fb4d5d8ffb2a940d5e2358 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfb4616fe5fb4d5d8ffb2a940d5e2358, State: Initialized, Role: FOLLOWER
I20260811 22:28:41.797120 24617 consensus_queue.cc:260] T d71c0b6bd70c42afa03645c32000f1ba P bfb4616fe5fb4d5d8ffb2a940d5e2358 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb4616fe5fb4d5d8ffb2a940d5e2358" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44241 } } peers { permanent_uuid: "7bf0fa1e88d04e0397be79b9099418d5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 46373 } } peers { permanent_uuid: "5470802c25f546c4b8955738575876b8" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41165 } }
I20260811 22:28:41.797989 24320 heartbeater.cc:503] Master 127.29.106.62:42607 was elected leader, sending a full tablet report...
I20260811 22:28:41.798475 24617 ts_tablet_manager.cc:1428] T d71c0b6bd70c42afa03645c32000f1ba P bfb4616fe5fb4d5d8ffb2a940d5e2358: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260811 22:28:41.843575 24625 raft_consensus.cc:482] T d71c0b6bd70c42afa03645c32000f1ba P bfb4616fe5fb4d5d8ffb2a940d5e2358 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:28:41.843827 24625 raft_consensus.cc:504] T d71c0b6bd70c42afa03645c32000f1ba P bfb4616fe5fb4d5d8ffb2a940d5e2358 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb4616fe5fb4d5d8ffb2a940d5e2358" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44241 } } peers { permanent_uuid: "7bf0fa1e88d04e0397be79b9099418d5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 46373 } } peers { permanent_uuid: "5470802c25f546c4b8955738575876b8" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41165 } }
I20260811 22:28:41.845297 24625 leader_election.cc:288] T d71c0b6bd70c42afa03645c32000f1ba P bfb4616fe5fb4d5d8ffb2a940d5e2358 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7bf0fa1e88d04e0397be79b9099418d5 (127.29.106.1:46373), 5470802c25f546c4b8955738575876b8 (127.29.106.3:41165)
I20260811 22:28:41.853737 24606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:41.845603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:47829 (local address 127.29.106.1:46373)
0811 22:28:41.845805 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:41.845810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:41.848422 (+ 2612us) server_negotiation.cc:407] Connection header received
0811 22:28:41.848735 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:41.848740 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:41.848818 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:41.848879 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:28:41.849512 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:41.850207 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:41.852069 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:41.852849 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:41.852993 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:41.853067 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:41.853363 (+ 296us) server_negotiation.cc:299] Negotiation successful
0811 22:28:41.853490 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260811 22:28:41.853854 24628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:41.845572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:46373 (local address 127.29.106.2:47829)
0811 22:28:41.848324 (+ 2752us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:41.848342 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 22:28:41.848540 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:28:41.849068 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:28:41.849071 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:28:41.849093 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:28:41.849378 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:41.849385 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:41.850340 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:41.850344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:41.851917 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:41.851933 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:41.853073 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:41.853079 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:41.853217 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:41.853243 (+ 26us) client_negotiation.cc:769] Sending connection context
0811 22:28:41.853326 (+ 83us) client_negotiation.cc:240] Negotiation successful
0811 22:28:41.853453 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2608,"mutex_wait_us":24,"thread_start_us":2411,"threads_started":1}
I20260811 22:28:41.853904 24610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:41.848496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:46861 (local address 127.29.106.3:41165)
0811 22:28:41.848674 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:41.848680 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:41.849048 (+ 368us) server_negotiation.cc:407] Connection header received
0811 22:28:41.849201 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:41.849206 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:41.849263 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:41.849334 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:28:41.849869 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:41.850586 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:41.852302 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:41.853037 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:41.853187 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:41.853256 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:41.853543 (+ 287us) server_negotiation.cc:299] Negotiation successful
0811 22:28:41.853666 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260811 22:28:41.854038 24629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:41.848396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.3:41165 (local address 127.29.106.2:46861)
0811 22:28:41.848949 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:41.848963 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 22:28:41.849080 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:28:41.849378 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:28:41.849382 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:28:41.849406 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:28:41.849733 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:41.849742 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:41.850730 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:41.850734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:41.852159 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:41.852171 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:41.853261 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:41.853268 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:41.853405 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:41.853427 (+ 22us) client_negotiation.cc:769] Sending connection context
0811 22:28:41.853504 (+ 77us) client_negotiation.cc:240] Negotiation successful
0811 22:28:41.853636 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"mutex_wait_us":3,"thread_start_us":305,"threads_started":1}
I20260811 22:28:41.854876 24498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d71c0b6bd70c42afa03645c32000f1ba" candidate_uuid: "bfb4616fe5fb4d5d8ffb2a940d5e2358" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5470802c25f546c4b8955738575876b8" is_pre_election: true
I20260811 22:28:41.854882 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d71c0b6bd70c42afa03645c32000f1ba" candidate_uuid: "bfb4616fe5fb4d5d8ffb2a940d5e2358" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bf0fa1e88d04e0397be79b9099418d5" is_pre_election: true
I20260811 22:28:41.855361 23977 raft_consensus.cc:2454] T d71c0b6bd70c42afa03645c32000f1ba P 7bf0fa1e88d04e0397be79b9099418d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfb4616fe5fb4d5d8ffb2a940d5e2358 in term 0.
I20260811 22:28:41.855361 24498 raft_consensus.cc:2454] T d71c0b6bd70c42afa03645c32000f1ba P 5470802c25f546c4b8955738575876b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfb4616fe5fb4d5d8ffb2a940d5e2358 in term 0.
I20260811 22:28:41.855934 24117 leader_election.cc:302] T d71c0b6bd70c42afa03645c32000f1ba P bfb4616fe5fb4d5d8ffb2a940d5e2358 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bf0fa1e88d04e0397be79b9099418d5, bfb4616fe5fb4d5d8ffb2a940d5e2358; no voters:
I20260811 22:28:41.856252 24625 raft_consensus.cc:2781] T d71c0b6bd70c42afa03645c32000f1ba P bfb4616fe5fb4d5d8ffb2a940d5e2358 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:28:41.856375 24625 raft_consensus.cc:482] T d71c0b6bd70c42afa03645c32000f1ba P bfb4616fe5fb4d5d8ffb2a940d5e2358 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:28:41.856475 24625 raft_consensus.cc:3037] T d71c0b6bd70c42afa03645c32000f1ba P bfb4616fe5fb4d5d8ffb2a940d5e2358 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:28:41.858605 24625 raft_consensus.cc:504] T d71c0b6bd70c42afa03645c32000f1ba P bfb4616fe5fb4d5d8ffb2a940d5e2358 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb4616fe5fb4d5d8ffb2a940d5e2358" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44241 } } peers { permanent_uuid: "7bf0fa1e88d04e0397be79b9099418d5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 46373 } } peers { permanent_uuid: "5470802c25f546c4b8955738575876b8" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41165 } }
I20260811 22:28:41.859118 24625 leader_election.cc:288] T d71c0b6bd70c42afa03645c32000f1ba P bfb4616fe5fb4d5d8ffb2a940d5e2358 [CANDIDATE]: Term 1 election: Requested vote from peers 7bf0fa1e88d04e0397be79b9099418d5 (127.29.106.1:46373), 5470802c25f546c4b8955738575876b8 (127.29.106.3:41165)
I20260811 22:28:41.859551 23977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d71c0b6bd70c42afa03645c32000f1ba" candidate_uuid: "bfb4616fe5fb4d5d8ffb2a940d5e2358" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bf0fa1e88d04e0397be79b9099418d5"
I20260811 22:28:41.859596 24498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d71c0b6bd70c42afa03645c32000f1ba" candidate_uuid: "bfb4616fe5fb4d5d8ffb2a940d5e2358" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5470802c25f546c4b8955738575876b8"
I20260811 22:28:41.859759 23977 raft_consensus.cc:3037] T d71c0b6bd70c42afa03645c32000f1ba P 7bf0fa1e88d04e0397be79b9099418d5 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:28:41.859760 24498 raft_consensus.cc:3037] T d71c0b6bd70c42afa03645c32000f1ba P 5470802c25f546c4b8955738575876b8 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:28:41.862900 24498 raft_consensus.cc:2454] T d71c0b6bd70c42afa03645c32000f1ba P 5470802c25f546c4b8955738575876b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfb4616fe5fb4d5d8ffb2a940d5e2358 in term 1.
I20260811 22:28:41.862900 23977 raft_consensus.cc:2454] T d71c0b6bd70c42afa03645c32000f1ba P 7bf0fa1e88d04e0397be79b9099418d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfb4616fe5fb4d5d8ffb2a940d5e2358 in term 1.
I20260811 22:28:41.863310 24117 leader_election.cc:302] T d71c0b6bd70c42afa03645c32000f1ba P bfb4616fe5fb4d5d8ffb2a940d5e2358 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bf0fa1e88d04e0397be79b9099418d5, bfb4616fe5fb4d5d8ffb2a940d5e2358; no voters:
I20260811 22:28:41.863572 24625 raft_consensus.cc:2781] T d71c0b6bd70c42afa03645c32000f1ba P bfb4616fe5fb4d5d8ffb2a940d5e2358 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:28:41.863981 24625 raft_consensus.cc:686] T d71c0b6bd70c42afa03645c32000f1ba P bfb4616fe5fb4d5d8ffb2a940d5e2358 [term 1 LEADER]: Becoming Leader. State: Replica: bfb4616fe5fb4d5d8ffb2a940d5e2358, State: Running, Role: LEADER
I20260811 22:28:41.864435 24625 consensus_queue.cc:237] T d71c0b6bd70c42afa03645c32000f1ba P bfb4616fe5fb4d5d8ffb2a940d5e2358 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb4616fe5fb4d5d8ffb2a940d5e2358" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44241 } } peers { permanent_uuid: "7bf0fa1e88d04e0397be79b9099418d5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 46373 } } peers { permanent_uuid: "5470802c25f546c4b8955738575876b8" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41165 } }
I20260811 22:28:41.868572 23697 catalog_manager.cc:5495] T d71c0b6bd70c42afa03645c32000f1ba P bfb4616fe5fb4d5d8ffb2a940d5e2358 reported cstate change: term changed from 0 to 1, leader changed from <none> to bfb4616fe5fb4d5d8ffb2a940d5e2358 (127.29.106.2). New cstate: current_term: 1 leader_uuid: "bfb4616fe5fb4d5d8ffb2a940d5e2358" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfb4616fe5fb4d5d8ffb2a940d5e2358" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44241 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7bf0fa1e88d04e0397be79b9099418d5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 46373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5470802c25f546c4b8955738575876b8" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41165 } health_report { overall_health: UNKNOWN } } }
I20260811 22:28:41.902374 24608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:41.897005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57934 (local address 127.29.106.2:44241)
0811 22:28:41.897200 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:41.897206 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:41.897327 (+ 121us) server_negotiation.cc:407] Connection header received
0811 22:28:41.897533 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:41.897537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:41.897621 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:41.897745 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:28:41.898126 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:41.898817 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:41.900332 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:41.900728 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:41.900836 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:41.901321 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:28:41.901819 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:28:41.901966 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:41.902049 (+ 83us) server_negotiation.cc:299] Negotiation successful
0811 22:28:41.902129 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260811 22:28:41.910112 24498 raft_consensus.cc:1261] T d71c0b6bd70c42afa03645c32000f1ba P 5470802c25f546c4b8955738575876b8 [term 1 FOLLOWER]: Refusing update from remote peer bfb4616fe5fb4d5d8ffb2a940d5e2358: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:28:41.910193 23977 raft_consensus.cc:1261] T d71c0b6bd70c42afa03645c32000f1ba P 7bf0fa1e88d04e0397be79b9099418d5 [term 1 FOLLOWER]: Refusing update from remote peer bfb4616fe5fb4d5d8ffb2a940d5e2358: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:28:41.911029 24630 consensus_queue.cc:1035] T d71c0b6bd70c42afa03645c32000f1ba P bfb4616fe5fb4d5d8ffb2a940d5e2358 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5470802c25f546c4b8955738575876b8" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:28:41.911427 24625 consensus_queue.cc:1035] T d71c0b6bd70c42afa03645c32000f1ba P bfb4616fe5fb4d5d8ffb2a940d5e2358 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bf0fa1e88d04e0397be79b9099418d5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 46373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260811 22:28:41.911523 24579 tablet.cc:2369] T d71c0b6bd70c42afa03645c32000f1ba P 5470802c25f546c4b8955738575876b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:28:41.921989 24635 mvcc.cc:205] Tried to move back new op lower bound from 7317452070534828032 to 7317452070360588288. Current Snapshot: MvccSnapshot[applied={T|T < 7317452070534828032}]
I20260811 22:28:41.923758 24636 mvcc.cc:205] Tried to move back new op lower bound from 7317452070534828032 to 7317452070360588288. Current Snapshot: MvccSnapshot[applied={T|T < 7317452070534828032}]
I20260811 22:28:41.936264 23696 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:28:41.936614 23696 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:28:41.940456 23696 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d: Sending DeleteTablet for 3 replicas of tablet d71c0b6bd70c42afa03645c32000f1ba
I20260811 22:28:41.941722 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 23820
I20260811 22:28:41.941656 24451 tablet_service.cc:1514] Processing DeleteTablet for tablet d71c0b6bd70c42afa03645c32000f1ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:28:41 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:55416
I20260811 22:28:41.941645 24198 tablet_service.cc:1514] Processing DeleteTablet for tablet d71c0b6bd70c42afa03645c32000f1ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:28:41 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:57926
I20260811 22:28:41.942526 23932 tablet_service.cc:1514] Processing DeleteTablet for tablet d71c0b6bd70c42afa03645c32000f1ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:28:41 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:60608
I20260811 22:28:41.966145 24662 tablet_replica.cc:330] T d71c0b6bd70c42afa03645c32000f1ba P 7bf0fa1e88d04e0397be79b9099418d5: stopping tablet replica
I20260811 22:28:41.966733 24662 raft_consensus.cc:2229] T d71c0b6bd70c42afa03645c32000f1ba P 7bf0fa1e88d04e0397be79b9099418d5 [term 1 FOLLOWER]: Raft consensus shutting down.
Aug 11 22:28:41 dist-test-slave-gjrz krb5kdc[23604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.1@KRBTEST.COM
I20260811 22:28:41.967490 24662 raft_consensus.cc:2258] T d71c0b6bd70c42afa03645c32000f1ba P 7bf0fa1e88d04e0397be79b9099418d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:28:41.968510 24657 tablet_replica.cc:330] T d71c0b6bd70c42afa03645c32000f1ba P 5470802c25f546c4b8955738575876b8: stopping tablet replica
I20260811 22:28:41.969079 24654 tablet_replica.cc:330] T d71c0b6bd70c42afa03645c32000f1ba P bfb4616fe5fb4d5d8ffb2a940d5e2358: stopping tablet replica
I20260811 22:28:41.969622 24657 raft_consensus.cc:2229] T d71c0b6bd70c42afa03645c32000f1ba P 5470802c25f546c4b8955738575876b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:28:41.970118 24654 raft_consensus.cc:2229] T d71c0b6bd70c42afa03645c32000f1ba P bfb4616fe5fb4d5d8ffb2a940d5e2358 [term 1 LEADER]: Raft consensus shutting down.
I20260811 22:28:41.970235 24662 ts_tablet_manager.cc:1905] T d71c0b6bd70c42afa03645c32000f1ba P 7bf0fa1e88d04e0397be79b9099418d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:28:41.970844 24654 raft_consensus.cc:2258] T d71c0b6bd70c42afa03645c32000f1ba P bfb4616fe5fb4d5d8ffb2a940d5e2358 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:28:41.971395 24657 raft_consensus.cc:2258] T d71c0b6bd70c42afa03645c32000f1ba P 5470802c25f546c4b8955738575876b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:28:41.974889 24657 ts_tablet_manager.cc:1905] T d71c0b6bd70c42afa03645c32000f1ba P 5470802c25f546c4b8955738575876b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:28:41.975791 24606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:41.942658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60612 (local address 127.29.106.1:46373)
0811 22:28:41.942830 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:41.942836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:41.955863 (+ 13027us) server_negotiation.cc:407] Connection header received
0811 22:28:41.956066 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:41.956071 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:41.956159 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:41.956264 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:28:41.958349 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:41.959676 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:41.961767 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:41.962508 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:41.962604 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:41.968415 (+ 5811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:28:41.968428 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:28:41.968444 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:28:41.968486 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:28:41.972236 (+ 3750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:41.973069 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:41.973078 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:41.973081 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:41.973234 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:41.973683 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:41.973688 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:41.973692 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:41.974157 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:28:41.974450 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:41.975095 (+ 645us) server_negotiation.cc:299] Negotiation successful
0811 22:28:41.975351 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260811 22:28:41.977175 24050 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:28:41.977306 24654 ts_tablet_manager.cc:1905] T d71c0b6bd70c42afa03645c32000f1ba P bfb4616fe5fb4d5d8ffb2a940d5e2358: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:28:41.986745 24657 ts_tablet_manager.cc:1918] T d71c0b6bd70c42afa03645c32000f1ba P 5470802c25f546c4b8955738575876b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:28:41.986984 24657 log.cc:1196] T d71c0b6bd70c42afa03645c32000f1ba P 5470802c25f546c4b8955738575876b8: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-2/wal/wals/d71c0b6bd70c42afa03645c32000f1ba
I20260811 22:28:41.987640 24657 ts_tablet_manager.cc:1939] T d71c0b6bd70c42afa03645c32000f1ba P 5470802c25f546c4b8955738575876b8: Deleting consensus metadata
I20260811 22:28:41.988536 24654 ts_tablet_manager.cc:1918] T d71c0b6bd70c42afa03645c32000f1ba P bfb4616fe5fb4d5d8ffb2a940d5e2358: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:28:41.988746 24654 log.cc:1196] T d71c0b6bd70c42afa03645c32000f1ba P bfb4616fe5fb4d5d8ffb2a940d5e2358: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-1/wal/wals/d71c0b6bd70c42afa03645c32000f1ba
I20260811 22:28:41.989312 24654 ts_tablet_manager.cc:1939] T d71c0b6bd70c42afa03645c32000f1ba P bfb4616fe5fb4d5d8ffb2a940d5e2358: Deleting consensus metadata
I20260811 22:28:41.989784 23679 catalog_manager.cc:4841] TS 5470802c25f546c4b8955738575876b8 (127.29.106.3:41165): tablet d71c0b6bd70c42afa03645c32000f1ba (table test-table [id=acfe615bb1124fe3a4c5f1d528269f47]) successfully deleted
I20260811 22:28:41.990985 23672 catalog_manager.cc:4841] TS bfb4616fe5fb4d5d8ffb2a940d5e2358 (127.29.106.2:44241): tablet d71c0b6bd70c42afa03645c32000f1ba (table test-table [id=acfe615bb1124fe3a4c5f1d528269f47]) successfully deleted
I20260811 22:28:42.069446 24662 ts_tablet_manager.cc:1918] T d71c0b6bd70c42afa03645c32000f1ba P 7bf0fa1e88d04e0397be79b9099418d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:28:42.069737 24662 log.cc:1196] T d71c0b6bd70c42afa03645c32000f1ba P 7bf0fa1e88d04e0397be79b9099418d5: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-0/wal/wals/d71c0b6bd70c42afa03645c32000f1ba
I20260811 22:28:42.070391 24662 ts_tablet_manager.cc:1939] T d71c0b6bd70c42afa03645c32000f1ba P 7bf0fa1e88d04e0397be79b9099418d5: Deleting consensus metadata
I20260811 22:28:42.070574 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 23820
I20260811 22:28:42.096423 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 24072
W20260811 22:28:42.096529 23679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.106.1:46373 (error 108)
Aug 11 22:28:42 dist-test-slave-gjrz krb5kdc[23604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.2@KRBTEST.COM
I20260811 22:28:42.116882 24604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:42.116348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:46373 (local address 127.0.0.1:60622)
0811 22:28:42.116537 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:42.116701 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.1:46373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260811 22:28:42.117267 23679 catalog_manager.cc:4584] TS 7bf0fa1e88d04e0397be79b9099418d5 (127.29.106.1:46373): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d71c0b6bd70c42afa03645c32000f1ba: Network error: Client connection negotiation failed: client connection to 127.29.106.1:46373: connect: Connection refused (error 111)
I20260811 22:28:42.117794 24608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:42.097898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57944 (local address 127.29.106.2:44241)
0811 22:28:42.098079 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:42.098085 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:42.098102 (+ 17us) server_negotiation.cc:407] Connection header received
0811 22:28:42.098176 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:42.098180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:42.098258 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:42.098345 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:28:42.100245 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:42.101489 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:42.103407 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:42.104071 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:42.104181 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:42.110059 (+ 5878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:28:42.110078 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:28:42.110098 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:28:42.110171 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:28:42.114104 (+ 3933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:42.114972 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:42.114981 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:42.114985 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:42.115168 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:42.115638 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:42.115642 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:42.115645 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:42.116188 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:28:42.116509 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:42.117268 (+ 759us) server_negotiation.cc:299] Negotiation successful
0811 22:28:42.117469 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260811 22:28:42.118460 24313 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:28:42.171020 24604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:42.170464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:46373 (local address 127.0.0.1:60632)
0811 22:28:42.170706 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:42.170820 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.1:46373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260811 22:28:42.172125 24604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:42.171734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:46373 (local address 127.0.0.1:60642)
0811 22:28:42.171876 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:42.171966 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.1:46373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260811 22:28:42.230608 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 24072
I20260811 22:28:42.232549 24604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:42.232035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:46373 (local address 127.0.0.1:60654)
0811 22:28:42.232230 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:42.232339 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.1:46373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260811 22:28:42.233968 24604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:42.233585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:46373 (local address 127.0.0.1:60666)
0811 22:28:42.233726 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:42.233814 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.1:46373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260811 22:28:42.256978 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 24334
Aug 11 22:28:42 dist-test-slave-gjrz krb5kdc[23604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.3@KRBTEST.COM
I20260811 22:28:42.271379 24610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:42.257675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55430 (local address 127.29.106.3:41165)
0811 22:28:42.257856 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:42.257862 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:42.257882 (+ 20us) server_negotiation.cc:407] Connection header received
0811 22:28:42.258011 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:42.258015 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:42.258080 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:42.258162 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:28:42.259532 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:42.260266 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:42.261542 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:42.261963 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:42.262074 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:42.266191 (+ 4117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:28:42.266206 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:28:42.266223 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:28:42.266256 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:28:42.268737 (+ 2481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:42.269374 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:42.269382 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:42.269384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:42.269533 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:42.269889 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:42.269892 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:42.269894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:42.270244 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:28:42.270519 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:42.270846 (+ 327us) server_negotiation.cc:299] Negotiation successful
0811 22:28:42.271063 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260811 22:28:42.272004 24573 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:28:42.314670 24604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:42.314211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:46373 (local address 127.0.0.1:60672)
0811 22:28:42.314414 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:42.314506 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.1:46373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260811 22:28:42.315776 24604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:42.315381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:46373 (local address 127.0.0.1:60688)
0811 22:28:42.315579 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:42.315652 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.1:46373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260811 22:28:42.359360 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 24334
I20260811 22:28:42.386965 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 23632
I20260811 22:28:42.388103 23748 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:28:42.497807 24604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:42.497062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:46373 (local address 127.0.0.1:60692)
0811 22:28:42.497401 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:42.497582 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.1:46373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169}
I20260811 22:28:42.497967 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 23632
I20260811 22:28:42.529214 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:42607
--webserver_interface=127.29.106.62
--webserver_port=43149
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:40907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:42607
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:28:42.854703 24674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:28:42.854970 24674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:28:42.855080 24674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:28:42.862082 24674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:28:42.862174 24674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:28:42.862227 24674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:28:42.862283 24674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:28:42.862327 24674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:28:42.872046 24674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:40907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:42607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:42607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.106.62
--webserver_port=43149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:28:42.873193 24674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:28:42.874549 24674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:28:42.899670 24687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:28:42.899693 24684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:28:42.899708 24685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:28:42.900516 24674 server_base.cc:1034] running on GCE node
I20260811 22:28:42.901335 24674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:28:42.902882 24674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:28:42.904091 24674 hybrid_clock.cc:648] HybridClock initialized: now 1786487322904032 us; error 42 us; skew 500 ppm
I20260811 22:28:42.904500 24674 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:28:42 dist-test-slave-gjrz krb5kdc[23604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487322, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:28:42.907932 24674 init.cc:377] Logged in from keytab as kudu/127.29.106.62@KRBTEST.COM (short username kudu)
I20260811 22:28:42.909593 24674 webserver.cc:460] Webserver started at http://127.29.106.62:43149/ using document root <none> and password file <none>
I20260811 22:28:42.910339 24674 fs_manager.cc:359] Metadata directory not provided
I20260811 22:28:42.910506 24674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:28:42.916420 24674 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260811 22:28:42.919449 24701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:28:42.920794 24674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260811 22:28:42.920985 24674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "d5ab3dad17d944cab5b01b0d6eff043d"
format_stamp: "Formatted at 2026-08-11 22:28:40 on dist-test-slave-gjrz"
server_key: "d44d0be8faa4eb8d8d25f9cdf3764516"
server_key_iv: "c7a0490464ba26dc465190ebf040657f"
server_key_version: "encryptionkey@0"
I20260811 22:28:42.922257 24674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:28:42.941422 24674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:28:42.945261 24674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:28:42.945513 24674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:28:42.963496 24674 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.62:42607
I20260811 22:28:42.963516 24796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.62:42607 every 8 connection(s)
I20260811 22:28:42.964972 24674 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:28:42.971755 24803 sys_catalog.cc:263] Verifying existing consensus state
I20260811 22:28:42.972645 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 24674
I20260811 22:28:42.973632 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.1:46373
--local_ip_for_outbound_sockets=127.29.106.1
--tserver_master_addrs=127.29.106.62:42607
--webserver_port=40377
--webserver_interface=127.29.106.1
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab
--principal=kudu/127.29.106.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:40907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 22:28:42.976271 24803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d: Bootstrap starting.
I20260811 22:28:42.998431 24803 log.cc:824] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d: Log is configured to *not* fsync() on all Append() calls
I20260811 22:28:43.010759 24803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 22:28:43.011284 24803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d: Bootstrap complete.
I20260811 22:28:43.020349 24803 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ab3dad17d944cab5b01b0d6eff043d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 42607 } }
I20260811 22:28:43.021142 24803 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5ab3dad17d944cab5b01b0d6eff043d, State: Initialized, Role: FOLLOWER
I20260811 22:28:43.021720 24803 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ab3dad17d944cab5b01b0d6eff043d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 42607 } }
I20260811 22:28:43.021962 24803 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:28:43.022089 24803 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:28:43.022248 24803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [term 1 FOLLOWER]: Advancing to term 2
I20260811 22:28:43.026136 24803 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ab3dad17d944cab5b01b0d6eff043d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 42607 } }
I20260811 22:28:43.026660 24803 leader_election.cc:302] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5ab3dad17d944cab5b01b0d6eff043d; no voters:
I20260811 22:28:43.027235 24803 leader_election.cc:288] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260811 22:28:43.027405 24810 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [term 2 FOLLOWER]: Leader election won for term 2
I20260811 22:28:43.029536 24803 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:28:43.031538 24810 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [term 2 LEADER]: Becoming Leader. State: Replica: d5ab3dad17d944cab5b01b0d6eff043d, State: Running, Role: LEADER
I20260811 22:28:43.032248 24810 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ab3dad17d944cab5b01b0d6eff043d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 42607 } }
I20260811 22:28:43.035607 24814 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5ab3dad17d944cab5b01b0d6eff043d. Latest consensus state: current_term: 2 leader_uuid: "d5ab3dad17d944cab5b01b0d6eff043d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ab3dad17d944cab5b01b0d6eff043d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 42607 } } }
I20260811 22:28:43.035959 24814 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [sys.catalog]: This master's current role is: LEADER
I20260811 22:28:43.036461 24812 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d5ab3dad17d944cab5b01b0d6eff043d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ab3dad17d944cab5b01b0d6eff043d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 42607 } } }
I20260811 22:28:43.036718 24812 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [sys.catalog]: This master's current role is: LEADER
I20260811 22:28:43.048215 24833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:28:43.055816 24833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:28:43.057257 24833 catalog_manager.cc:1259] Loaded cluster ID: a4952745047642089b4a0fcfdfcfd184
I20260811 22:28:43.057411 24833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:28:43.060632 24833 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260811 22:28:43.078456 24833 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260811 22:28:43.087110 24833 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:28:43.088806 24833 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d: Loaded TSK: 0
I20260811 22:28:43.090243 24833 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260811 22:28:43.320287 24806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:28:43.320564 24806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:28:43.320698 24806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:28:43.327955 24806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:28:43.328063 24806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:28:43.328203 24806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.1
I20260811 22:28:43.337834 24806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:40907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab
--principal=kudu/127.29.106.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.1:46373
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.106.1
--webserver_port=40377
--tserver_master_addrs=127.29.106.62:42607
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.1
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:28:43.338976 24806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:28:43.340276 24806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:28:43.352989 24861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:28:43.353035 24868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:28:43.353070 24863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:28:43.354103 24806 server_base.cc:1034] running on GCE node
I20260811 22:28:43.354696 24806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:28:43.355580 24806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:28:43.356786 24806 hybrid_clock.cc:648] HybridClock initialized: now 1786487323356724 us; error 53 us; skew 500 ppm
I20260811 22:28:43.357065 24806 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:28:43 dist-test-slave-gjrz krb5kdc[23604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487323, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:28:43.360348 24806 init.cc:377] Logged in from keytab as kudu/127.29.106.1@KRBTEST.COM (short username kudu)
I20260811 22:28:43.361977 24806 webserver.cc:460] Webserver started at http://127.29.106.1:40377/ using document root <none> and password file <none>
I20260811 22:28:43.362820 24806 fs_manager.cc:359] Metadata directory not provided
I20260811 22:28:43.362977 24806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:28:43.369426 24806 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260811 22:28:43.372498 24879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:28:43.373919 24806 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:28:43.374081 24806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-0/wal
uuid: "7bf0fa1e88d04e0397be79b9099418d5"
format_stamp: "Formatted at 2026-08-11 22:28:40 on dist-test-slave-gjrz"
server_key: "56282eed7170f61813e443af894ab68d"
server_key_iv: "f0ff4839e78c33dba7882b0666c8e859"
server_key_version: "encryptionkey@0"
I20260811 22:28:43.375380 24806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:28:43.403007 24806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:28:43.408001 24806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:28:43.408288 24806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:28:43.409395 24806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:28:43.410948 24806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:28:43.411054 24806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:28:43.411157 24806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:28:43.411234 24806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:28:43.442713 24806 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.1:46373
I20260811 22:28:43.442781 25074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.1:46373 every 8 connection(s)
I20260811 22:28:43.444123 24806 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
I20260811 22:28:43.451802 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 24806
I20260811 22:28:43.452526 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.2:44241
--local_ip_for_outbound_sockets=127.29.106.2
--tserver_master_addrs=127.29.106.62:42607
--webserver_port=40101
--webserver_interface=127.29.106.2
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab
--principal=kudu/127.29.106.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:40907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 22:28:43 dist-test-slave-gjrz krb5kdc[23604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487323, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.1@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:28:43.467336 25081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:43.447066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.1:51265)
0811 22:28:43.447919 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:43.447974 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0811 22:28:43.449667 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:28:43.450872 (+ 1205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:28:43.450879 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:28:43.451543 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:28:43.452290 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:43.452315 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:43.453965 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:43.453971 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:43.455689 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:43.455704 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:43.456675 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:43.456681 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:43.456936 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:43.457581 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:28:43.457615 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:28:43.460028 (+ 2413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:28:43.463247 (+ 3219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:28:43.463256 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:28:43.463290 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:28:43.463714 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:28:43.464143 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:28:43.464148 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:28:43.464152 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:28:43.464332 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:28:43.465035 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:28:43.465042 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:28:43.465351 (+ 309us) client_negotiation.cc:769] Sending connection context
0811 22:28:43.465577 (+ 226us) client_negotiation.cc:240] Negotiation successful
0811 22:28:43.466097 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":361,"threads_started":1}
I20260811 22:28:43.467350 25077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:43.446797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:51265 (local address 127.29.106.62:42607)
0811 22:28:43.447381 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:43.447393 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:43.448102 (+ 709us) server_negotiation.cc:407] Connection header received
0811 22:28:43.450148 (+ 2046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:43.450160 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:43.450460 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:43.450742 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:28:43.452507 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:43.453793 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:43.455811 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:43.456447 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:43.456584 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:43.460305 (+ 3721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:28:43.460312 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:28:43.460325 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:28:43.460361 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:28:43.463111 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:43.463853 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:43.463860 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:43.463866 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:43.464025 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:43.464447 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:43.464450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:43.464452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:43.464899 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:28:43.465085 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:43.465820 (+ 735us) server_negotiation.cc:299] Negotiation successful
0811 22:28:43.466195 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":232,"threads_started":1}
I20260811 22:28:43.469983 25075 heartbeater.cc:347] Connected to a master server at 127.29.106.62:42607
I20260811 22:28:43.470283 25075 heartbeater.cc:464] Registering TS with master...
I20260811 22:28:43.471135 25075 heartbeater.cc:511] Master 127.29.106.62:42607 requested a full tablet report, sending...
I20260811 22:28:43.473268 24729 ts_manager.cc:194] Registered new tserver with Master: 7bf0fa1e88d04e0397be79b9099418d5 (127.29.106.1:46373)
I20260811 22:28:43.474723 24729 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.1@KRBTEST.COM'} at 127.29.106.1:51265
W20260811 22:28:43.777109 25082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:28:43.777365 25082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:28:43.777486 25082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:28:43.784555 25082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:28:43.784647 25082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:28:43.784792 25082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.2
I20260811 22:28:43.793823 25082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:40907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab
--principal=kudu/127.29.106.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.2:44241
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.106.2
--webserver_port=40101
--tserver_master_addrs=127.29.106.62:42607
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.2
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:28:43.794917 25082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:28:43.796207 25082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:28:43.808655 25093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:28:43.808588 25092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:28:43.808689 25099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:28:43.809482 25082 server_base.cc:1034] running on GCE node
I20260811 22:28:43.810074 25082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:28:43.810923 25082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:28:43.812127 25082 hybrid_clock.cc:648] HybridClock initialized: now 1786487323812073 us; error 45 us; skew 500 ppm
I20260811 22:28:43.812412 25082 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:28:43 dist-test-slave-gjrz krb5kdc[23604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487323, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:28:43.815330 25082 init.cc:377] Logged in from keytab as kudu/127.29.106.2@KRBTEST.COM (short username kudu)
I20260811 22:28:43.816599 25082 webserver.cc:460] Webserver started at http://127.29.106.2:40101/ using document root <none> and password file <none>
I20260811 22:28:43.817252 25082 fs_manager.cc:359] Metadata directory not provided
I20260811 22:28:43.817366 25082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:28:43.822274 25082 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260811 22:28:43.824783 25111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:28:43.826025 25082 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260811 22:28:43.826160 25082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-1/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-1/wal
uuid: "bfb4616fe5fb4d5d8ffb2a940d5e2358"
format_stamp: "Formatted at 2026-08-11 22:28:41 on dist-test-slave-gjrz"
server_key: "4f13b02c6438febf776ce8441d9de1b4"
server_key_iv: "5bd54651f209888a484ba02d8f8abbc2"
server_key_version: "encryptionkey@0"
I20260811 22:28:43.827107 25082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:28:43.837394 25082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:28:43.840823 25082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:28:43.841034 25082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:28:43.841869 25082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:28:43.843065 25082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:28:43.843152 25082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:28:43.843227 25082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:28:43.843318 25082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:28:43.875097 25082 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.2:44241
I20260811 22:28:43.875208 25310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.2:44241 every 8 connection(s)
I20260811 22:28:43.876510 25082 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
I20260811 22:28:43.878243 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 25082
I20260811 22:28:43.878893 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.3:41165
--local_ip_for_outbound_sockets=127.29.106.3
--tserver_master_addrs=127.29.106.62:42607
--webserver_port=45753
--webserver_interface=127.29.106.3
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab
--principal=kudu/127.29.106.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:40907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 22:28:43 dist-test-slave-gjrz krb5kdc[23604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487323, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.2@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:28:43.896231 25077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:43.879160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:57205 (local address 127.29.106.62:42607)
0811 22:28:43.879390 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:43.879395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:43.880841 (+ 1446us) server_negotiation.cc:407] Connection header received
0811 22:28:43.882716 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:43.882722 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:43.882806 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:43.882913 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:28:43.884367 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:43.885309 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:43.886852 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:43.887453 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:43.887579 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:43.890885 (+ 3306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:28:43.890892 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:28:43.890896 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:28:43.890930 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:28:43.893318 (+ 2388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:43.894098 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:43.894101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:43.894103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:43.894220 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:43.894636 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:43.894639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:43.894641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:43.894879 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:28:43.894999 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:43.895758 (+ 759us) server_negotiation.cc:299] Negotiation successful
0811 22:28:43.895932 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260811 22:28:43.897478 25313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:43.879824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.2:57205)
0811 22:28:43.880648 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:43.880701 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0811 22:28:43.882448 (+ 1747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:28:43.883101 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:28:43.883108 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:28:43.883657 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:28:43.884196 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:43.884213 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:43.885497 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:43.885501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:43.886702 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:43.886715 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:43.887664 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:43.887669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:43.887861 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:43.888464 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:28:43.888488 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:28:43.890685 (+ 2197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:28:43.893461 (+ 2776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:28:43.893471 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:28:43.893486 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:28:43.894008 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:28:43.894308 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:28:43.894312 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:28:43.894316 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:28:43.894547 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:28:43.895013 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:28:43.895020 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:28:43.895404 (+ 384us) client_negotiation.cc:769] Sending connection context
0811 22:28:43.895674 (+ 270us) client_negotiation.cc:240] Negotiation successful
0811 22:28:43.896144 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":342,"threads_started":1}
I20260811 22:28:43.898842 25311 heartbeater.cc:347] Connected to a master server at 127.29.106.62:42607
I20260811 22:28:43.899152 25311 heartbeater.cc:464] Registering TS with master...
I20260811 22:28:43.899991 25311 heartbeater.cc:511] Master 127.29.106.62:42607 requested a full tablet report, sending...
I20260811 22:28:43.901453 24729 ts_manager.cc:194] Registered new tserver with Master: bfb4616fe5fb4d5d8ffb2a940d5e2358 (127.29.106.2:44241)
I20260811 22:28:43.902297 24729 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.2@KRBTEST.COM'} at 127.29.106.2:57205
W20260811 22:28:44.207216 25317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:28:44.207491 25317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:28:44.207628 25317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:28:44.214514 25317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:28:44.214603 25317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:28:44.214744 25317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.3
I20260811 22:28:44.223822 25317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:40907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab
--principal=kudu/127.29.106.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.3:41165
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.106.3
--webserver_port=45753
--tserver_master_addrs=127.29.106.62:42607
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.3
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:28:44.224933 25317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:28:44.226202 25317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:28:44.238480 25334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:28:44.238480 25338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:28:44.238722 25331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:28:44.239117 25317 server_base.cc:1034] running on GCE node
I20260811 22:28:44.239709 25317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:28:44.240558 25317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:28:44.241753 25317 hybrid_clock.cc:648] HybridClock initialized: now 1786487324241716 us; error 24 us; skew 500 ppm
I20260811 22:28:44.242051 25317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:28:44 dist-test-slave-gjrz krb5kdc[23604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487324, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:28:44.244966 25317 init.cc:377] Logged in from keytab as kudu/127.29.106.3@KRBTEST.COM (short username kudu)
I20260811 22:28:44.246251 25317 webserver.cc:460] Webserver started at http://127.29.106.3:45753/ using document root <none> and password file <none>
I20260811 22:28:44.246865 25317 fs_manager.cc:359] Metadata directory not provided
I20260811 22:28:44.246996 25317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:28:44.252015 25317 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260811 22:28:44.254479 25350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:28:44.255679 25317 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:28:44.255828 25317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-2/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-2/wal
uuid: "5470802c25f546c4b8955738575876b8"
format_stamp: "Formatted at 2026-08-11 22:28:41 on dist-test-slave-gjrz"
server_key: "8b6b27dc3bd88b4a88c6ae6fcaa0015a"
server_key_iv: "69684c349d7dd5247f2bddfd3b31e38b"
server_key_version: "encryptionkey@0"
I20260811 22:28:44.256733 25317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:28:44.280303 25317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:28:44.284016 25317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:28:44.284262 25317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:28:44.285175 25317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:28:44.286458 25317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:28:44.286556 25317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:28:44.286661 25317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:28:44.286731 25317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:28:44.316982 25317 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.3:41165
I20260811 22:28:44.317049 25542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.3:41165 every 8 connection(s)
I20260811 22:28:44.318343 25317 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
I20260811 22:28:44.319519 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 25317
Aug 11 22:28:44 dist-test-slave-gjrz krb5kdc[23604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487324, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.3@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:28:44.336130 25077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:44.320579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42588 (local address 127.29.106.62:42607)
0811 22:28:44.320767 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:44.320774 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:44.321244 (+ 470us) server_negotiation.cc:407] Connection header received
0811 22:28:44.321442 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:44.321447 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:44.321537 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:44.321665 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:28:44.323548 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:44.324403 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:44.326382 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:44.326921 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:44.327072 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:44.330431 (+ 3359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:28:44.330438 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:28:44.330443 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:28:44.330486 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:28:44.333416 (+ 2930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:44.334040 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:44.334046 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:44.334049 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:44.334212 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:44.334596 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:44.334601 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:44.334605 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:44.334957 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:28:44.335224 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:44.335579 (+ 355us) server_negotiation.cc:299] Negotiation successful
0811 22:28:44.335788 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260811 22:28:44.340306 25552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:44.321119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:49273 (local address 127.29.106.62:42607)
0811 22:28:44.322853 (+ 1734us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:44.322857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:44.322873 (+ 16us) server_negotiation.cc:407] Connection header received
0811 22:28:44.324757 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:44.324761 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:44.324814 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:44.324890 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:28:44.326741 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:44.327705 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:44.329968 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:44.330577 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:44.330718 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:44.334167 (+ 3449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:28:44.334174 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:28:44.334178 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:28:44.334209 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:28:44.337283 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:44.338027 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:44.338032 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:44.338035 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:44.338186 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:44.338601 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:44.338605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:44.338608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:44.338928 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:28:44.339072 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:44.339744 (+ 672us) server_negotiation.cc:299] Negotiation successful
0811 22:28:44.339943 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1648,"thread_start_us":411,"threads_started":1}
I20260811 22:28:44.341156 25555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:44.321702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.3:49273)
0811 22:28:44.322565 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:44.322612 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0811 22:28:44.324486 (+ 1874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:28:44.325142 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:28:44.325153 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:28:44.325859 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:28:44.326574 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:44.326595 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:44.327851 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:44.327857 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:44.329806 (+ 1949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:44.329822 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:44.330802 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:44.330810 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:44.331087 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:44.331700 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:28:44.331722 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:28:44.333958 (+ 2236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:28:44.337423 (+ 3465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:28:44.337430 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:28:44.337443 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:28:44.337923 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:28:44.338296 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:28:44.338299 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:28:44.338301 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:28:44.338483 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:28:44.339060 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:28:44.339068 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:28:44.339414 (+ 346us) client_negotiation.cc:769] Sending connection context
0811 22:28:44.339648 (+ 234us) client_negotiation.cc:240] Negotiation successful
0811 22:28:44.340141 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":318,"threads_started":1}
I20260811 22:28:44.342368 25546 heartbeater.cc:347] Connected to a master server at 127.29.106.62:42607
I20260811 22:28:44.342612 25546 heartbeater.cc:464] Registering TS with master...
I20260811 22:28:44.343254 25546 heartbeater.cc:511] Master 127.29.106.62:42607 requested a full tablet report, sending...
I20260811 22:28:44.344508 24729 ts_manager.cc:194] Registered new tserver with Master: 5470802c25f546c4b8955738575876b8 (127.29.106.3:41165)
I20260811 22:28:44.345278 24729 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.3@KRBTEST.COM'} at 127.29.106.3:49273
I20260811 22:28:44.350822 30120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 22:28:44.356838 25552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:44.351761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42592 (local address 127.29.106.62:42607)
0811 22:28:44.351895 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:44.351900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:44.352314 (+ 414us) server_negotiation.cc:407] Connection header received
0811 22:28:44.352468 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:44.352472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:44.352607 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:44.352718 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:28:44.353055 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:44.353720 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:44.355049 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:44.355470 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:44.355562 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:44.355986 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:28:44.356431 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:28:44.356530 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:44.356600 (+ 70us) server_negotiation.cc:299] Negotiation successful
0811 22:28:44.356657 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260811 22:28:44.357682 24729 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:42592:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:28:44.364599 24729 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:28:44.400688 25576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:44.375579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:46373 (local address 127.0.0.1:60700)
0811 22:28:44.388725 (+ 13146us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:44.388769 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 22:28:44.388945 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:28:44.391389 (+ 2444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:28:44.391397 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:28:44.391445 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:28:44.391943 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:44.391957 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:44.395018 (+ 3061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:44.395025 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:44.397788 (+ 2763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:44.397808 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:44.399683 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:44.399690 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:44.399907 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:44.399969 (+ 62us) client_negotiation.cc:769] Sending connection context
0811 22:28:44.400096 (+ 127us) client_negotiation.cc:240] Negotiation successful
0811 22:28:44.400270 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12858,"mutex_wait_us":73,"spinlock_wait_cycles":5504,"thread_start_us":15089,"threads_started":1}
I20260811 22:28:44.400691 25574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:44.375799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60700 (local address 127.29.106.1:46373)
0811 22:28:44.390471 (+ 14672us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:44.390480 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:44.390507 (+ 27us) server_negotiation.cc:407] Connection header received
0811 22:28:44.390627 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:44.390635 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:44.390885 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:44.391075 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:28:44.393193 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:44.394816 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:44.398004 (+ 3188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:44.399249 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:44.399415 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:44.399636 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:44.400255 (+ 619us) server_negotiation.cc:299] Negotiation successful
0811 22:28:44.400419 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14523,"thread_start_us":14416,"threads_started":1}
I20260811 22:28:44.401160 25577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:44.376584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57960 (local address 127.29.106.2:44241)
0811 22:28:44.390208 (+ 13624us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:44.390218 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:44.390556 (+ 338us) server_negotiation.cc:407] Connection header received
0811 22:28:44.390711 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:44.390716 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:44.390887 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:44.391080 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:28:44.392313 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:44.393995 (+ 1682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:44.398244 (+ 4249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:44.399796 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:44.399923 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:44.400124 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:44.400426 (+ 302us) server_negotiation.cc:299] Negotiation successful
0811 22:28:44.400600 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13431,"thread_start_us":13271,"threads_started":1}
I20260811 22:28:44.401919 25572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:44.376130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.2:44241 (local address 127.0.0.1:57960)
0811 22:28:44.390443 (+ 14313us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:44.390461 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 22:28:44.390587 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:28:44.391392 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:28:44.391397 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:28:44.391466 (+ 69us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:28:44.392065 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:44.392079 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:44.394198 (+ 2119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:44.394204 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:44.398054 (+ 3850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:44.398069 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:44.400026 (+ 1957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:44.400031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:44.400190 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:44.400224 (+ 34us) client_negotiation.cc:769] Sending connection context
0811 22:28:44.401415 (+ 1191us) client_negotiation.cc:240] Negotiation successful
0811 22:28:44.401545 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14193,"mutex_wait_us":1,"spinlock_wait_cycles":1792,"thread_start_us":12994,"threads_started":1}
I20260811 22:28:44.404723 24961 tablet_service.cc:1467] Processing CreateTablet for tablet 32dc157bfc984a00a62ff4768ef032ac (DEFAULT_TABLE table=test-table [id=85adc91b885e42588e3c207913a16c6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:28:44.404932 25194 tablet_service.cc:1467] Processing CreateTablet for tablet 32dc157bfc984a00a62ff4768ef032ac (DEFAULT_TABLE table=test-table [id=85adc91b885e42588e3c207913a16c6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:28:44.406193 24961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32dc157bfc984a00a62ff4768ef032ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:28:44.406232 25194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32dc157bfc984a00a62ff4768ef032ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:28:44.407156 25578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:44.391016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.3:41165 (local address 127.0.0.1:55432)
0811 22:28:44.398022 (+ 7006us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:44.398040 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 22:28:44.398144 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:28:44.399291 (+ 1147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:28:44.399296 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:28:44.399323 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:28:44.399738 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:44.399749 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:44.402577 (+ 2828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:44.402582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:44.404602 (+ 2020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:44.404617 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:44.406448 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:44.406455 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:44.406627 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:44.406659 (+ 32us) client_negotiation.cc:769] Sending connection context
0811 22:28:44.406748 (+ 89us) client_negotiation.cc:240] Negotiation successful
0811 22:28:44.406866 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6861,"thread_start_us":6674,"threads_started":1}
I20260811 22:28:44.407351 25579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:44.391397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55432 (local address 127.29.106.3:41165)
0811 22:28:44.397783 (+ 6386us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:44.397791 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:44.398746 (+ 955us) server_negotiation.cc:407] Connection header received
0811 22:28:44.398849 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:44.398858 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:44.399077 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:44.399282 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:28:44.399921 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:44.402414 (+ 2493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:44.404775 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:44.406154 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:44.406292 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:44.406504 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:44.406826 (+ 322us) server_negotiation.cc:299] Negotiation successful
0811 22:28:44.406992 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6155,"thread_start_us":5942,"threads_started":1}
I20260811 22:28:44.410856 25437 tablet_service.cc:1467] Processing CreateTablet for tablet 32dc157bfc984a00a62ff4768ef032ac (DEFAULT_TABLE table=test-table [id=85adc91b885e42588e3c207913a16c6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:28:44.412149 25437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32dc157bfc984a00a62ff4768ef032ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:28:44.417714 25580 tablet_bootstrap.cc:492] T 32dc157bfc984a00a62ff4768ef032ac P bfb4616fe5fb4d5d8ffb2a940d5e2358: Bootstrap starting.
I20260811 22:28:44.420845 25581 tablet_bootstrap.cc:492] T 32dc157bfc984a00a62ff4768ef032ac P 7bf0fa1e88d04e0397be79b9099418d5: Bootstrap starting.
I20260811 22:28:44.421113 25580 tablet_bootstrap.cc:654] T 32dc157bfc984a00a62ff4768ef032ac P bfb4616fe5fb4d5d8ffb2a940d5e2358: Neither blocks nor log segments found. Creating new log.
I20260811 22:28:44.422480 25580 log.cc:824] T 32dc157bfc984a00a62ff4768ef032ac P bfb4616fe5fb4d5d8ffb2a940d5e2358: Log is configured to *not* fsync() on all Append() calls
I20260811 22:28:44.424546 25581 tablet_bootstrap.cc:654] T 32dc157bfc984a00a62ff4768ef032ac P 7bf0fa1e88d04e0397be79b9099418d5: Neither blocks nor log segments found. Creating new log.
I20260811 22:28:44.425851 25580 tablet_bootstrap.cc:492] T 32dc157bfc984a00a62ff4768ef032ac P bfb4616fe5fb4d5d8ffb2a940d5e2358: No bootstrap required, opened a new log
I20260811 22:28:44.425837 25581 log.cc:824] T 32dc157bfc984a00a62ff4768ef032ac P 7bf0fa1e88d04e0397be79b9099418d5: Log is configured to *not* fsync() on all Append() calls
I20260811 22:28:44.426131 25580 ts_tablet_manager.cc:1397] T 32dc157bfc984a00a62ff4768ef032ac P bfb4616fe5fb4d5d8ffb2a940d5e2358: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260811 22:28:44.428952 25581 tablet_bootstrap.cc:492] T 32dc157bfc984a00a62ff4768ef032ac P 7bf0fa1e88d04e0397be79b9099418d5: No bootstrap required, opened a new log
I20260811 22:28:44.429211 25581 ts_tablet_manager.cc:1397] T 32dc157bfc984a00a62ff4768ef032ac P 7bf0fa1e88d04e0397be79b9099418d5: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260811 22:28:44.427297 25587 tablet_bootstrap.cc:492] T 32dc157bfc984a00a62ff4768ef032ac P 5470802c25f546c4b8955738575876b8: Bootstrap starting.
I20260811 22:28:44.434163 25587 tablet_bootstrap.cc:654] T 32dc157bfc984a00a62ff4768ef032ac P 5470802c25f546c4b8955738575876b8: Neither blocks nor log segments found. Creating new log.
I20260811 22:28:44.435050 25587 log.cc:824] T 32dc157bfc984a00a62ff4768ef032ac P 5470802c25f546c4b8955738575876b8: Log is configured to *not* fsync() on all Append() calls
I20260811 22:28:44.436354 25580 raft_consensus.cc:348] T 32dc157bfc984a00a62ff4768ef032ac P bfb4616fe5fb4d5d8ffb2a940d5e2358 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bf0fa1e88d04e0397be79b9099418d5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 46373 } } peers { permanent_uuid: "5470802c25f546c4b8955738575876b8" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41165 } } peers { permanent_uuid: "bfb4616fe5fb4d5d8ffb2a940d5e2358" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44241 } }
I20260811 22:28:44.436784 25580 raft_consensus.cc:374] T 32dc157bfc984a00a62ff4768ef032ac P bfb4616fe5fb4d5d8ffb2a940d5e2358 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:28:44.436885 25580 raft_consensus.cc:729] T 32dc157bfc984a00a62ff4768ef032ac P bfb4616fe5fb4d5d8ffb2a940d5e2358 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfb4616fe5fb4d5d8ffb2a940d5e2358, State: Initialized, Role: FOLLOWER
I20260811 22:28:44.437566 25580 consensus_queue.cc:260] T 32dc157bfc984a00a62ff4768ef032ac P bfb4616fe5fb4d5d8ffb2a940d5e2358 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bf0fa1e88d04e0397be79b9099418d5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 46373 } } peers { permanent_uuid: "5470802c25f546c4b8955738575876b8" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41165 } } peers { permanent_uuid: "bfb4616fe5fb4d5d8ffb2a940d5e2358" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44241 } }
I20260811 22:28:44.437892 25587 tablet_bootstrap.cc:492] T 32dc157bfc984a00a62ff4768ef032ac P 5470802c25f546c4b8955738575876b8: No bootstrap required, opened a new log
I20260811 22:28:44.438143 25587 ts_tablet_manager.cc:1397] T 32dc157bfc984a00a62ff4768ef032ac P 5470802c25f546c4b8955738575876b8: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20260811 22:28:44.439584 25581 raft_consensus.cc:348] T 32dc157bfc984a00a62ff4768ef032ac P 7bf0fa1e88d04e0397be79b9099418d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bf0fa1e88d04e0397be79b9099418d5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 46373 } } peers { permanent_uuid: "5470802c25f546c4b8955738575876b8" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41165 } } peers { permanent_uuid: "bfb4616fe5fb4d5d8ffb2a940d5e2358" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44241 } }
I20260811 22:28:44.439922 25311 heartbeater.cc:503] Master 127.29.106.62:42607 was elected leader, sending a full tablet report...
I20260811 22:28:44.439987 25581 raft_consensus.cc:374] T 32dc157bfc984a00a62ff4768ef032ac P 7bf0fa1e88d04e0397be79b9099418d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:28:44.440086 25581 raft_consensus.cc:729] T 32dc157bfc984a00a62ff4768ef032ac P 7bf0fa1e88d04e0397be79b9099418d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bf0fa1e88d04e0397be79b9099418d5, State: Initialized, Role: FOLLOWER
I20260811 22:28:44.440691 25580 ts_tablet_manager.cc:1428] T 32dc157bfc984a00a62ff4768ef032ac P bfb4616fe5fb4d5d8ffb2a940d5e2358: Time spent starting tablet: real 0.014s user 0.007s sys 0.007s
I20260811 22:28:44.440745 25581 consensus_queue.cc:260] T 32dc157bfc984a00a62ff4768ef032ac P 7bf0fa1e88d04e0397be79b9099418d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bf0fa1e88d04e0397be79b9099418d5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 46373 } } peers { permanent_uuid: "5470802c25f546c4b8955738575876b8" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41165 } } peers { permanent_uuid: "bfb4616fe5fb4d5d8ffb2a940d5e2358" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44241 } }
I20260811 22:28:44.441836 25075 heartbeater.cc:503] Master 127.29.106.62:42607 was elected leader, sending a full tablet report...
I20260811 22:28:44.442369 25581 ts_tablet_manager.cc:1428] T 32dc157bfc984a00a62ff4768ef032ac P 7bf0fa1e88d04e0397be79b9099418d5: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
W20260811 22:28:44.447507 25076 tablet.cc:2369] T 32dc157bfc984a00a62ff4768ef032ac P 7bf0fa1e88d04e0397be79b9099418d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:28:44.448964 25587 raft_consensus.cc:348] T 32dc157bfc984a00a62ff4768ef032ac P 5470802c25f546c4b8955738575876b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bf0fa1e88d04e0397be79b9099418d5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 46373 } } peers { permanent_uuid: "5470802c25f546c4b8955738575876b8" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41165 } } peers { permanent_uuid: "bfb4616fe5fb4d5d8ffb2a940d5e2358" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44241 } }
I20260811 22:28:44.449361 25587 raft_consensus.cc:374] T 32dc157bfc984a00a62ff4768ef032ac P 5470802c25f546c4b8955738575876b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:28:44.449456 25587 raft_consensus.cc:729] T 32dc157bfc984a00a62ff4768ef032ac P 5470802c25f546c4b8955738575876b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5470802c25f546c4b8955738575876b8, State: Initialized, Role: FOLLOWER
I20260811 22:28:44.449980 25587 consensus_queue.cc:260] T 32dc157bfc984a00a62ff4768ef032ac P 5470802c25f546c4b8955738575876b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bf0fa1e88d04e0397be79b9099418d5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 46373 } } peers { permanent_uuid: "5470802c25f546c4b8955738575876b8" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41165 } } peers { permanent_uuid: "bfb4616fe5fb4d5d8ffb2a940d5e2358" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44241 } }
I20260811 22:28:44.450870 25546 heartbeater.cc:503] Master 127.29.106.62:42607 was elected leader, sending a full tablet report...
I20260811 22:28:44.451421 25587 ts_tablet_manager.cc:1428] T 32dc157bfc984a00a62ff4768ef032ac P 5470802c25f546c4b8955738575876b8: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
W20260811 22:28:44.569972 25550 tablet.cc:2369] T 32dc157bfc984a00a62ff4768ef032ac P 5470802c25f546c4b8955738575876b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 22:28:44.628350 25312 tablet.cc:2369] T 32dc157bfc984a00a62ff4768ef032ac P bfb4616fe5fb4d5d8ffb2a940d5e2358: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:28:44.757980 25597 raft_consensus.cc:482] T 32dc157bfc984a00a62ff4768ef032ac P 5470802c25f546c4b8955738575876b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:28:44.758260 25597 raft_consensus.cc:504] T 32dc157bfc984a00a62ff4768ef032ac P 5470802c25f546c4b8955738575876b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bf0fa1e88d04e0397be79b9099418d5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 46373 } } peers { permanent_uuid: "5470802c25f546c4b8955738575876b8" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41165 } } peers { permanent_uuid: "bfb4616fe5fb4d5d8ffb2a940d5e2358" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44241 } }
I20260811 22:28:44.759967 25597 leader_election.cc:288] T 32dc157bfc984a00a62ff4768ef032ac P 5470802c25f546c4b8955738575876b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7bf0fa1e88d04e0397be79b9099418d5 (127.29.106.1:46373), bfb4616fe5fb4d5d8ffb2a940d5e2358 (127.29.106.2:44241)
I20260811 22:28:44.766798 25574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:44.760305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:55011 (local address 127.29.106.1:46373)
0811 22:28:44.760502 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:44.760508 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:44.760833 (+ 325us) server_negotiation.cc:407] Connection header received
0811 22:28:44.761012 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:44.761016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:44.761094 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:44.761196 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:28:44.761556 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:44.762527 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:44.764918 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:44.766038 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:44.766202 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:44.766277 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:44.766473 (+ 196us) server_negotiation.cc:299] Negotiation successful
0811 22:28:44.766605 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260811 22:28:44.766870 25555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:44.760388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:46373 (local address 127.29.106.3:55011)
0811 22:28:44.760699 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:44.760719 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 22:28:44.760862 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:28:44.761217 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:28:44.761220 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:28:44.761254 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:28:44.761411 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:44.761421 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:44.762658 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:44.762661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:44.764778 (+ 2117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:44.764794 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:44.766181 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:44.766185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:44.766328 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:44.766353 (+ 25us) client_negotiation.cc:769] Sending connection context
0811 22:28:44.766434 (+ 81us) client_negotiation.cc:240] Negotiation successful
0811 22:28:44.766578 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":90}
I20260811 22:28:44.767208 25577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:44.760539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:56689 (local address 127.29.106.2:44241)
0811 22:28:44.760771 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:44.760776 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:44.761081 (+ 305us) server_negotiation.cc:407] Connection header received
0811 22:28:44.761213 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:44.761217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:44.761290 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:44.761380 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:28:44.761959 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:44.762862 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:44.765171 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:44.766348 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:44.766473 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:44.766539 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:44.766860 (+ 321us) server_negotiation.cc:299] Negotiation successful
0811 22:28:44.766986 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260811 22:28:44.767578 25602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:44.760429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.2:44241 (local address 127.29.106.3:56689)
0811 22:28:44.760981 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:44.760997 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:28:44.761099 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:28:44.761404 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:28:44.761408 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:28:44.761429 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:28:44.761788 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:44.761798 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:44.762997 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:44.763002 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:44.765030 (+ 2028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:28:44.765049 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:28:44.766528 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:28:44.766534 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:28:44.766698 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:44.766730 (+ 32us) client_negotiation.cc:769] Sending connection context
0811 22:28:44.766815 (+ 85us) client_negotiation.cc:240] Negotiation successful
0811 22:28:44.767235 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"mutex_wait_us":35,"thread_start_us":295,"threads_started":1}
I20260811 22:28:44.767892 25001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32dc157bfc984a00a62ff4768ef032ac" candidate_uuid: "5470802c25f546c4b8955738575876b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bf0fa1e88d04e0397be79b9099418d5" is_pre_election: true
I20260811 22:28:44.768482 25001 raft_consensus.cc:2454] T 32dc157bfc984a00a62ff4768ef032ac P 7bf0fa1e88d04e0397be79b9099418d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5470802c25f546c4b8955738575876b8 in term 0.
I20260811 22:28:44.768563 25238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32dc157bfc984a00a62ff4768ef032ac" candidate_uuid: "5470802c25f546c4b8955738575876b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfb4616fe5fb4d5d8ffb2a940d5e2358" is_pre_election: true
I20260811 22:28:44.769030 25238 raft_consensus.cc:2454] T 32dc157bfc984a00a62ff4768ef032ac P bfb4616fe5fb4d5d8ffb2a940d5e2358 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5470802c25f546c4b8955738575876b8 in term 0.
I20260811 22:28:44.769213 25354 leader_election.cc:302] T 32dc157bfc984a00a62ff4768ef032ac P 5470802c25f546c4b8955738575876b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5470802c25f546c4b8955738575876b8, 7bf0fa1e88d04e0397be79b9099418d5; no voters:
I20260811 22:28:44.769574 25597 raft_consensus.cc:2781] T 32dc157bfc984a00a62ff4768ef032ac P 5470802c25f546c4b8955738575876b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:28:44.769701 25597 raft_consensus.cc:482] T 32dc157bfc984a00a62ff4768ef032ac P 5470802c25f546c4b8955738575876b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:28:44.769798 25597 raft_consensus.cc:3037] T 32dc157bfc984a00a62ff4768ef032ac P 5470802c25f546c4b8955738575876b8 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:28:44.771999 25597 raft_consensus.cc:504] T 32dc157bfc984a00a62ff4768ef032ac P 5470802c25f546c4b8955738575876b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bf0fa1e88d04e0397be79b9099418d5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 46373 } } peers { permanent_uuid: "5470802c25f546c4b8955738575876b8" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41165 } } peers { permanent_uuid: "bfb4616fe5fb4d5d8ffb2a940d5e2358" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44241 } }
I20260811 22:28:44.772513 25597 leader_election.cc:288] T 32dc157bfc984a00a62ff4768ef032ac P 5470802c25f546c4b8955738575876b8 [CANDIDATE]: Term 1 election: Requested vote from peers 7bf0fa1e88d04e0397be79b9099418d5 (127.29.106.1:46373), bfb4616fe5fb4d5d8ffb2a940d5e2358 (127.29.106.2:44241)
I20260811 22:28:44.772897 25001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32dc157bfc984a00a62ff4768ef032ac" candidate_uuid: "5470802c25f546c4b8955738575876b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bf0fa1e88d04e0397be79b9099418d5"
I20260811 22:28:44.772924 25238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32dc157bfc984a00a62ff4768ef032ac" candidate_uuid: "5470802c25f546c4b8955738575876b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfb4616fe5fb4d5d8ffb2a940d5e2358"
I20260811 22:28:44.773064 25001 raft_consensus.cc:3037] T 32dc157bfc984a00a62ff4768ef032ac P 7bf0fa1e88d04e0397be79b9099418d5 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:28:44.773092 25238 raft_consensus.cc:3037] T 32dc157bfc984a00a62ff4768ef032ac P bfb4616fe5fb4d5d8ffb2a940d5e2358 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:28:44.775130 25001 raft_consensus.cc:2454] T 32dc157bfc984a00a62ff4768ef032ac P 7bf0fa1e88d04e0397be79b9099418d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5470802c25f546c4b8955738575876b8 in term 1.
I20260811 22:28:44.775192 25238 raft_consensus.cc:2454] T 32dc157bfc984a00a62ff4768ef032ac P bfb4616fe5fb4d5d8ffb2a940d5e2358 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5470802c25f546c4b8955738575876b8 in term 1.
I20260811 22:28:44.775511 25354 leader_election.cc:302] T 32dc157bfc984a00a62ff4768ef032ac P 5470802c25f546c4b8955738575876b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5470802c25f546c4b8955738575876b8, 7bf0fa1e88d04e0397be79b9099418d5; no voters:
I20260811 22:28:44.775796 25597 raft_consensus.cc:2781] T 32dc157bfc984a00a62ff4768ef032ac P 5470802c25f546c4b8955738575876b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:28:44.776206 25597 raft_consensus.cc:686] T 32dc157bfc984a00a62ff4768ef032ac P 5470802c25f546c4b8955738575876b8 [term 1 LEADER]: Becoming Leader. State: Replica: 5470802c25f546c4b8955738575876b8, State: Running, Role: LEADER
I20260811 22:28:44.776682 25597 consensus_queue.cc:237] T 32dc157bfc984a00a62ff4768ef032ac P 5470802c25f546c4b8955738575876b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bf0fa1e88d04e0397be79b9099418d5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 46373 } } peers { permanent_uuid: "5470802c25f546c4b8955738575876b8" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41165 } } peers { permanent_uuid: "bfb4616fe5fb4d5d8ffb2a940d5e2358" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44241 } }
I20260811 22:28:44.780992 24729 catalog_manager.cc:5495] T 32dc157bfc984a00a62ff4768ef032ac P 5470802c25f546c4b8955738575876b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5470802c25f546c4b8955738575876b8 (127.29.106.3). New cstate: current_term: 1 leader_uuid: "5470802c25f546c4b8955738575876b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bf0fa1e88d04e0397be79b9099418d5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 46373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5470802c25f546c4b8955738575876b8" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bfb4616fe5fb4d5d8ffb2a940d5e2358" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44241 } health_report { overall_health: UNKNOWN } } }
I20260811 22:28:44.836742 25579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:44.831661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55442 (local address 127.29.106.3:41165)
0811 22:28:44.831857 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:44.831864 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:44.831964 (+ 100us) server_negotiation.cc:407] Connection header received
0811 22:28:44.832138 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:44.832141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:44.832218 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:44.832345 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:28:44.832672 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:44.833390 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:44.834770 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:44.835158 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:44.835245 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:44.835669 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:28:44.836171 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:28:44.836301 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:44.836412 (+ 111us) server_negotiation.cc:299] Negotiation successful
0811 22:28:44.836488 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260811 22:28:44.844343 25001 raft_consensus.cc:1261] T 32dc157bfc984a00a62ff4768ef032ac P 7bf0fa1e88d04e0397be79b9099418d5 [term 1 FOLLOWER]: Refusing update from remote peer 5470802c25f546c4b8955738575876b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:28:44.844337 25238 raft_consensus.cc:1261] T 32dc157bfc984a00a62ff4768ef032ac P bfb4616fe5fb4d5d8ffb2a940d5e2358 [term 1 FOLLOWER]: Refusing update from remote peer 5470802c25f546c4b8955738575876b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:28:44.845252 25597 consensus_queue.cc:1035] T 32dc157bfc984a00a62ff4768ef032ac P 5470802c25f546c4b8955738575876b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bf0fa1e88d04e0397be79b9099418d5" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 46373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:28:44.845531 25603 consensus_queue.cc:1035] T 32dc157bfc984a00a62ff4768ef032ac P 5470802c25f546c4b8955738575876b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfb4616fe5fb4d5d8ffb2a940d5e2358" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:28:44.870103 24729 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:42592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:28:44.870332 24729 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:42592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:28:44.873500 24729 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d: Sending DeleteTablet for 3 replicas of tablet 32dc157bfc984a00a62ff4768ef032ac
I20260811 22:28:44.874625 24961 tablet_service.cc:1514] Processing DeleteTablet for tablet 32dc157bfc984a00a62ff4768ef032ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:28:44 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:60700
I20260811 22:28:44.874735 25194 tablet_service.cc:1514] Processing DeleteTablet for tablet 32dc157bfc984a00a62ff4768ef032ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:28:44 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:57960
I20260811 22:28:44.879540 25437 tablet_service.cc:1514] Processing DeleteTablet for tablet 32dc157bfc984a00a62ff4768ef032ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:28:44 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:55432
I20260811 22:28:44.887115 25637 tablet_replica.cc:330] T 32dc157bfc984a00a62ff4768ef032ac P bfb4616fe5fb4d5d8ffb2a940d5e2358: stopping tablet replica
I20260811 22:28:44.888003 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 24806
I20260811 22:28:44.888502 25637 raft_consensus.cc:2229] T 32dc157bfc984a00a62ff4768ef032ac P bfb4616fe5fb4d5d8ffb2a940d5e2358 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:28:44.889837 25637 raft_consensus.cc:2258] T 32dc157bfc984a00a62ff4768ef032ac P bfb4616fe5fb4d5d8ffb2a940d5e2358 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:28:44.892943 25637 ts_tablet_manager.cc:1905] T 32dc157bfc984a00a62ff4768ef032ac P bfb4616fe5fb4d5d8ffb2a940d5e2358: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:28:44.895212 25638 tablet_replica.cc:330] T 32dc157bfc984a00a62ff4768ef032ac P 5470802c25f546c4b8955738575876b8: stopping tablet replica
I20260811 22:28:44.895628 25638 raft_consensus.cc:2229] T 32dc157bfc984a00a62ff4768ef032ac P 5470802c25f546c4b8955738575876b8 [term 1 LEADER]: Raft consensus shutting down.
I20260811 22:28:44.896379 25638 raft_consensus.cc:2258] T 32dc157bfc984a00a62ff4768ef032ac P 5470802c25f546c4b8955738575876b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:28:44.897198 25635 tablet_replica.cc:330] T 32dc157bfc984a00a62ff4768ef032ac P 7bf0fa1e88d04e0397be79b9099418d5: stopping tablet replica
I20260811 22:28:44.897598 25635 raft_consensus.cc:2229] T 32dc157bfc984a00a62ff4768ef032ac P 7bf0fa1e88d04e0397be79b9099418d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:28:44.898095 25635 raft_consensus.cc:2258] T 32dc157bfc984a00a62ff4768ef032ac P 7bf0fa1e88d04e0397be79b9099418d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:28:44.900203 25635 ts_tablet_manager.cc:1905] T 32dc157bfc984a00a62ff4768ef032ac P 7bf0fa1e88d04e0397be79b9099418d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:28:44.903558 25637 ts_tablet_manager.cc:1918] T 32dc157bfc984a00a62ff4768ef032ac P bfb4616fe5fb4d5d8ffb2a940d5e2358: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:28:44.903793 25637 log.cc:1196] T 32dc157bfc984a00a62ff4768ef032ac P bfb4616fe5fb4d5d8ffb2a940d5e2358: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-1/wal/wals/32dc157bfc984a00a62ff4768ef032ac
I20260811 22:28:44.904399 25638 ts_tablet_manager.cc:1905] T 32dc157bfc984a00a62ff4768ef032ac P 5470802c25f546c4b8955738575876b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:28:44.904498 25637 ts_tablet_manager.cc:1939] T 32dc157bfc984a00a62ff4768ef032ac P bfb4616fe5fb4d5d8ffb2a940d5e2358: Deleting consensus metadata
I20260811 22:28:44.906913 25635 ts_tablet_manager.cc:1918] T 32dc157bfc984a00a62ff4768ef032ac P 7bf0fa1e88d04e0397be79b9099418d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:28:44.907063 25635 log.cc:1196] T 32dc157bfc984a00a62ff4768ef032ac P 7bf0fa1e88d04e0397be79b9099418d5: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-0/wal/wals/32dc157bfc984a00a62ff4768ef032ac
I20260811 22:28:44.906980 24702 catalog_manager.cc:4841] TS bfb4616fe5fb4d5d8ffb2a940d5e2358 (127.29.106.2:44241): tablet 32dc157bfc984a00a62ff4768ef032ac (table test-table [id=85adc91b885e42588e3c207913a16c6e]) successfully deleted
I20260811 22:28:44.907841 25635 ts_tablet_manager.cc:1939] T 32dc157bfc984a00a62ff4768ef032ac P 7bf0fa1e88d04e0397be79b9099418d5: Deleting consensus metadata
I20260811 22:28:44.909528 24705 catalog_manager.cc:4841] TS 7bf0fa1e88d04e0397be79b9099418d5 (127.29.106.1:46373): tablet 32dc157bfc984a00a62ff4768ef032ac (table test-table [id=85adc91b885e42588e3c207913a16c6e]) successfully deleted
I20260811 22:28:44.910992 25638 ts_tablet_manager.cc:1918] T 32dc157bfc984a00a62ff4768ef032ac P 5470802c25f546c4b8955738575876b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:28:44.911151 25638 log.cc:1196] T 32dc157bfc984a00a62ff4768ef032ac P 5470802c25f546c4b8955738575876b8: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-2/wal/wals/32dc157bfc984a00a62ff4768ef032ac
I20260811 22:28:44.911755 25638 ts_tablet_manager.cc:1939] T 32dc157bfc984a00a62ff4768ef032ac P 5470802c25f546c4b8955738575876b8: Deleting consensus metadata
I20260811 22:28:44.913281 24705 catalog_manager.cc:4841] TS 5470802c25f546c4b8955738575876b8 (127.29.106.3:41165): tablet 32dc157bfc984a00a62ff4768ef032ac (table test-table [id=85adc91b885e42588e3c207913a16c6e]) successfully deleted
I20260811 22:28:44.924633 25574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:44.889156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60716 (local address 127.29.106.1:46373)
0811 22:28:44.889346 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:44.889353 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:44.904619 (+ 15266us) server_negotiation.cc:407] Connection header received
0811 22:28:44.904836 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:44.904841 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:44.904937 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:44.905045 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:28:44.908066 (+ 3021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:44.909151 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:44.912110 (+ 2959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:44.912718 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:44.912831 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:44.919102 (+ 6271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:28:44.919114 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:28:44.919127 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:28:44.919162 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:28:44.921841 (+ 2679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:44.922537 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:44.922545 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:44.922547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:44.922681 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:44.923065 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:44.923068 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:44.923071 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:44.923483 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:28:44.923756 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:44.924114 (+ 358us) server_negotiation.cc:299] Negotiation successful
0811 22:28:44.924330 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260811 22:28:44.925248 25061 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:28:45.027935 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 24806
I20260811 22:28:45.056775 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 25082
I20260811 22:28:45.075944 25577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:45.057612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57972 (local address 127.29.106.2:44241)
0811 22:28:45.057784 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:45.057789 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:45.057811 (+ 22us) server_negotiation.cc:407] Connection header received
0811 22:28:45.058009 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:45.058013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:45.058090 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:45.058169 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:28:45.060092 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:45.060913 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:45.062834 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:45.063249 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:45.063391 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:45.070185 (+ 6794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:28:45.070202 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:28:45.070218 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:28:45.070255 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:28:45.073077 (+ 2822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:45.073909 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:45.073917 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:45.073919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:45.074064 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:45.074438 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:45.074441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:45.074443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:45.074807 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:28:45.075020 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:45.075435 (+ 415us) server_negotiation.cc:299] Negotiation successful
0811 22:28:45.075614 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260811 22:28:45.076637 25305 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:28:45.165449 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 25082
I20260811 22:28:45.198338 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 25317
I20260811 22:28:45.214447 25579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:45.199096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55444 (local address 127.29.106.3:41165)
0811 22:28:45.199315 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0811 22:28:45.199320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:28:45.199422 (+ 102us) server_negotiation.cc:407] Connection header received
0811 22:28:45.199635 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:28:45.199639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:28:45.199703 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:28:45.199788 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:28:45.201752 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:45.202656 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:45.204545 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:28:45.205002 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:28:45.205107 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:28:45.208531 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:28:45.208557 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:28:45.208576 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:28:45.208610 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:28:45.211624 (+ 3014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:45.212446 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:45.212453 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:45.212455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:45.212625 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:28:45.213011 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:28:45.213014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:28:45.213016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:28:45.213357 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:28:45.213616 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0811 22:28:45.213950 (+ 334us) server_negotiation.cc:299] Negotiation successful
0811 22:28:45.214112 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260811 22:28:45.215157 25537 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:28:45.341038 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 25317
I20260811 22:28:45.369279 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 24674
I20260811 22:28:45.370458 24791 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:28:45.482280 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 24674
I20260811 22:28:45.513898 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:42607
--webserver_interface=127.29.106.62
--webserver_port=43149
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:40907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:42607
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:28:45.838869 25649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:28:45.839125 25649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:28:45.839257 25649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:28:45.846242 25649 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:28:45.846334 25649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:28:45.846405 25649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:28:45.846489 25649 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:28:45.846544 25649 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:28:45.855890 25649 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:40907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:42607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:42607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.106.62
--webserver_port=43149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:28:45.857103 25649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:28:45.858448 25649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:28:45.869598 25661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:28:45.869539 25664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:28:45.869630 25660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:28:45.870558 25649 server_base.cc:1034] running on GCE node
I20260811 22:28:45.871245 25649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:28:45.872591 25649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:28:45.873785 25649 hybrid_clock.cc:648] HybridClock initialized: now 1786487325873725 us; error 49 us; skew 500 ppm
I20260811 22:28:45.874076 25649 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:28:45 dist-test-slave-gjrz krb5kdc[23604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487325, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:28:45.877156 25649 init.cc:377] Logged in from keytab as kudu/127.29.106.62@KRBTEST.COM (short username kudu)
I20260811 22:28:45.878610 25649 webserver.cc:460] Webserver started at http://127.29.106.62:43149/ using document root <none> and password file <none>
I20260811 22:28:45.879292 25649 fs_manager.cc:359] Metadata directory not provided
I20260811 22:28:45.879426 25649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:28:45.884709 25649 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260811 22:28:45.887457 25679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:28:45.888669 25649 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260811 22:28:45.888826 25649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "d5ab3dad17d944cab5b01b0d6eff043d"
format_stamp: "Formatted at 2026-08-11 22:28:40 on dist-test-slave-gjrz"
server_key: "d44d0be8faa4eb8d8d25f9cdf3764516"
server_key_iv: "c7a0490464ba26dc465190ebf040657f"
server_key_version: "encryptionkey@0"
I20260811 22:28:45.889756 25649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:28:45.906471 25649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:28:45.910197 25649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:28:45.910427 25649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:28:45.928272 25649 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.62:42607
I20260811 22:28:45.928292 25767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.62:42607 every 8 connection(s)
I20260811 22:28:45.929682 25649 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:28:45.936403 25768 sys_catalog.cc:263] Verifying existing consensus state
I20260811 22:28:45.937851 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 25649
I20260811 22:28:45.938664 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.1:46373
--local_ip_for_outbound_sockets=127.29.106.1
--tserver_master_addrs=127.29.106.62:42607
--webserver_port=40377
--webserver_interface=127.29.106.1
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab
--principal=kudu/127.29.106.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:40907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 22:28:45.939597 25768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d: Bootstrap starting.
I20260811 22:28:45.962914 25768 log.cc:824] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d: Log is configured to *not* fsync() on all Append() calls
I20260811 22:28:45.984555 25768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 22:28:45.985185 25768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d: Bootstrap complete.
I20260811 22:28:45.998513 25768 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ab3dad17d944cab5b01b0d6eff043d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 42607 } }
I20260811 22:28:45.999739 25768 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5ab3dad17d944cab5b01b0d6eff043d, State: Initialized, Role: FOLLOWER
I20260811 22:28:46.000470 25768 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ab3dad17d944cab5b01b0d6eff043d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 42607 } }
I20260811 22:28:46.000744 25768 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:28:46.000864 25768 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:28:46.001063 25768 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [term 2 FOLLOWER]: Advancing to term 3
I20260811 22:28:46.006573 25768 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ab3dad17d944cab5b01b0d6eff043d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 42607 } }
I20260811 22:28:46.007182 25768 leader_election.cc:302] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5ab3dad17d944cab5b01b0d6eff043d; no voters:
I20260811 22:28:46.007869 25768 leader_election.cc:288] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260811 22:28:46.007995 25777 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [term 3 FOLLOWER]: Leader election won for term 3
I20260811 22:28:46.009229 25777 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [term 3 LEADER]: Becoming Leader. State: Replica: d5ab3dad17d944cab5b01b0d6eff043d, State: Running, Role: LEADER
I20260811 22:28:46.009799 25777 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ab3dad17d944cab5b01b0d6eff043d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 42607 } }
I20260811 22:28:46.010274 25768 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:28:46.012209 25779 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5ab3dad17d944cab5b01b0d6eff043d. Latest consensus state: current_term: 3 leader_uuid: "d5ab3dad17d944cab5b01b0d6eff043d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ab3dad17d944cab5b01b0d6eff043d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 42607 } } }
I20260811 22:28:46.012142 25778 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d5ab3dad17d944cab5b01b0d6eff043d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ab3dad17d944cab5b01b0d6eff043d" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 42607 } } }
I20260811 22:28:46.012473 25778 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [sys.catalog]: This master's current role is: LEADER
I20260811 22:28:46.012477 25779 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5ab3dad17d944cab5b01b0d6eff043d [sys.catalog]: This master's current role is: LEADER
I20260811 22:28:46.026412 25783 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:28:46.037954 25783 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:28:46.039549 25783 catalog_manager.cc:1259] Loaded cluster ID: a4952745047642089b4a0fcfdfcfd184
I20260811 22:28:46.039700 25783 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260811 22:28:46.041849 25783 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f3fbfca09ad google::LogMessage::Fail() at ??:0
@ 0x7f3fbfca388a google::LogMessage::SendToLog() at ??:0
@ 0x7f3fbfca04de google::LogMessage::Flush() at ??:0
@ 0x7f3fbfca1a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f3fd56d3512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
I20260811 22:28:46.055586 25808 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f3fd56d0976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f3fd5758610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f3fd5758439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f3fc28ec345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f3fc28ef91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f3fc28ef799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f3fc28caeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f3fc5ba46db start_thread at ??:0
@ 0x7f3fbeb3a61f clone at ??:0
W20260811 22:28:46.361294 25773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:28:46.361544 25773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:28:46.361665 25773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:28:46.368719 25773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:28:46.368839 25773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:28:46.369011 25773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.1
I20260811 22:28:46.378268 25773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:40907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab
--principal=kudu/127.29.106.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.1:46373
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.106.1
--webserver_port=40377
--tserver_master_addrs=127.29.106.62:42607
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.1
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:28:46.379400 25773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:28:46.380626 25773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:28:46.393110 25852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:28:46.393113 25857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:28:46.393119 25849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:28:46.394101 25773 server_base.cc:1034] running on GCE node
I20260811 22:28:46.394668 25773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:28:46.395526 25773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:28:46.396742 25773 hybrid_clock.cc:648] HybridClock initialized: now 1786487326396689 us; error 26 us; skew 500 ppm
I20260811 22:28:46.397027 25773 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:28:46 dist-test-slave-gjrz krb5kdc[23604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487326, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:28:46.399993 25773 init.cc:377] Logged in from keytab as kudu/127.29.106.1@KRBTEST.COM (short username kudu)
I20260811 22:28:46.401324 25773 webserver.cc:460] Webserver started at http://127.29.106.1:40377/ using document root <none> and password file <none>
I20260811 22:28:46.401950 25773 fs_manager.cc:359] Metadata directory not provided
I20260811 22:28:46.402068 25773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:28:46.407056 25773 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:28:46.409682 25868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:28:46.410871 25773 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 22:28:46.411021 25773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-0/wal
uuid: "7bf0fa1e88d04e0397be79b9099418d5"
format_stamp: "Formatted at 2026-08-11 22:28:40 on dist-test-slave-gjrz"
server_key: "56282eed7170f61813e443af894ab68d"
server_key_iv: "f0ff4839e78c33dba7882b0666c8e859"
server_key_version: "encryptionkey@0"
I20260811 22:28:46.412004 25773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:28:46.428323 25773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:28:46.431775 25773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:28:46.432008 25773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:28:46.432904 25773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:28:46.434144 25773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:28:46.434270 25773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:28:46.434377 25773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:28:46.434456 25773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:28:46.465044 25773 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.1:46373
I20260811 22:28:46.465111 26089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.1:46373 every 8 connection(s)
I20260811 22:28:46.466398 25773 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
I20260811 22:28:46.470770 26092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:46.469071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.1:32935)
0811 22:28:46.469779 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:46.469994 (+ 215us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":278,"threads_started":1}
W20260811 22:28:46.471396 26090 heartbeater.cc:644] Failed to heartbeat to 127.29.106.62:42607 (0 consecutive failures): Network error: Failed to ping master at 127.29.106.62:42607: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
I20260811 22:28:46.472080 26092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:46.471789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.1:57559)
0811 22:28:46.471902 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:46.471964 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260811 22:28:46.472901 26092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:46.472547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.1:44029)
0811 22:28:46.472680 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:46.472766 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260811 22:28:46.472919 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 25773
W20260811 22:28:46.473191 26090 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260811 22:28:46.473604 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.2:44241
--local_ip_for_outbound_sockets=127.29.106.2
--tserver_master_addrs=127.29.106.62:42607
--webserver_port=40101
--webserver_interface=127.29.106.2
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab
--principal=kudu/127.29.106.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:40907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:28:46.797878 26093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:28:46.798113 26093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:28:46.798241 26093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:28:46.805307 26093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:28:46.805401 26093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:28:46.805550 26093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.2
I20260811 22:28:46.814601 26093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:40907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab
--principal=kudu/127.29.106.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.2:44241
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.106.2
--webserver_port=40101
--tserver_master_addrs=127.29.106.62:42607
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.2
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:28:46.815737 26093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:28:46.816984 26093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:28:46.829226 26101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:28:46.829185 26103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:28:46.829249 26107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:28:46.830049 26093 server_base.cc:1034] running on GCE node
I20260811 22:28:46.830565 26093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:28:46.831447 26093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:28:46.832643 26093 hybrid_clock.cc:648] HybridClock initialized: now 1786487326832604 us; error 54 us; skew 500 ppm
I20260811 22:28:46.832913 26093 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:28:46 dist-test-slave-gjrz krb5kdc[23604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487326, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:28:46.835757 26093 init.cc:377] Logged in from keytab as kudu/127.29.106.2@KRBTEST.COM (short username kudu)
I20260811 22:28:46.837050 26093 webserver.cc:460] Webserver started at http://127.29.106.2:40101/ using document root <none> and password file <none>
I20260811 22:28:46.837616 26093 fs_manager.cc:359] Metadata directory not provided
I20260811 22:28:46.837715 26093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:28:46.842562 26093 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260811 22:28:46.845038 26121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:28:46.846204 26093 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 22:28:46.846352 26093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-1/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-1/wal
uuid: "bfb4616fe5fb4d5d8ffb2a940d5e2358"
format_stamp: "Formatted at 2026-08-11 22:28:41 on dist-test-slave-gjrz"
server_key: "4f13b02c6438febf776ce8441d9de1b4"
server_key_iv: "5bd54651f209888a484ba02d8f8abbc2"
server_key_version: "encryptionkey@0"
I20260811 22:28:46.847338 26093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:28:46.868300 26093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:28:46.871783 26093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:28:46.872021 26093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:28:46.872927 26093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:28:46.874207 26093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:28:46.874303 26093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:28:46.874404 26093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:28:46.874475 26093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:28:46.904325 26093 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.2:44241
I20260811 22:28:46.904399 26342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.2:44241 every 8 connection(s)
I20260811 22:28:46.905683 26093 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
I20260811 22:28:46.907606 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 26093
I20260811 22:28:46.908191 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.3:41165
--local_ip_for_outbound_sockets=127.29.106.3
--tserver_master_addrs=127.29.106.62:42607
--webserver_port=45753
--webserver_interface=127.29.106.3
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab
--principal=kudu/127.29.106.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:40907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 22:28:46.910468 26348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:46.908553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.2:46817)
0811 22:28:46.909333 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:46.909613 (+ 280us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":306,"threads_started":1}
W20260811 22:28:46.911060 26345 heartbeater.cc:644] Failed to heartbeat to 127.29.106.62:42607 (0 consecutive failures): Network error: Failed to ping master at 127.29.106.62:42607: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
I20260811 22:28:46.911839 26348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:46.911551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.2:41187)
0811 22:28:46.911672 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:46.911736 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260811 22:28:46.912603 26348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:46.912335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.2:57797)
0811 22:28:46.912439 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:46.912504 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260811 22:28:46.912818 26345 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260811 22:28:47.232456 26349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:28:47.232703 26349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:28:47.232803 26349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:28:47.239763 26349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:28:47.239857 26349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:28:47.239972 26349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.3
I20260811 22:28:47.249176 26349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:40907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab
--principal=kudu/127.29.106.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.3:41165
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.106.3
--webserver_port=45753
--tserver_master_addrs=127.29.106.62:42607
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.3
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:28:47.250377 26349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:28:47.251688 26349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:28:47.264115 26360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:28:47.264070 26361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:28:47.264154 26363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:28:47.265040 26349 server_base.cc:1034] running on GCE node
I20260811 22:28:47.265564 26349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:28:47.266377 26349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:28:47.267568 26349 hybrid_clock.cc:648] HybridClock initialized: now 1786487327267522 us; error 39 us; skew 500 ppm
I20260811 22:28:47.267835 26349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:28:47 dist-test-slave-gjrz krb5kdc[23604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487327, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:28:47.270803 26349 init.cc:377] Logged in from keytab as kudu/127.29.106.3@KRBTEST.COM (short username kudu)
I20260811 22:28:47.272152 26349 webserver.cc:460] Webserver started at http://127.29.106.3:45753/ using document root <none> and password file <none>
I20260811 22:28:47.272816 26349 fs_manager.cc:359] Metadata directory not provided
I20260811 22:28:47.272921 26349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:28:47.277942 26349 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260811 22:28:47.280490 26377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:28:47.281729 26349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:28:47.281857 26349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-2/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-2/wal
uuid: "5470802c25f546c4b8955738575876b8"
format_stamp: "Formatted at 2026-08-11 22:28:41 on dist-test-slave-gjrz"
server_key: "8b6b27dc3bd88b4a88c6ae6fcaa0015a"
server_key_iv: "69684c349d7dd5247f2bddfd3b31e38b"
server_key_version: "encryptionkey@0"
I20260811 22:28:47.282822 26349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:28:47.298357 26349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:28:47.301925 26349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:28:47.302139 26349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:28:47.303025 26349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:28:47.304275 26349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:28:47.304368 26349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:28:47.304448 26349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:28:47.304538 26349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:28:47.334911 26349 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.3:41165
I20260811 22:28:47.334976 26567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.3:41165 every 8 connection(s)
I20260811 22:28:47.336251 26349 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
I20260811 22:28:47.340692 26570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:47.338935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.3:57487)
0811 22:28:47.339684 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:47.339908 (+ 224us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":300,"threads_started":1}
W20260811 22:28:47.341279 26568 heartbeater.cc:644] Failed to heartbeat to 127.29.106.62:42607 (0 consecutive failures): Network error: Failed to ping master at 127.29.106.62:42607: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
I20260811 22:28:47.341980 26570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:47.341669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.3:58297)
0811 22:28:47.341783 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:47.341844 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260811 22:28:47.342695 26570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:47.342410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.3:54627)
0811 22:28:47.342525 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:47.342596 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260811 22:28:47.342917 26568 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260811 22:28:47.343613 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 26349
I20260811 22:28:47.474503 26575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:47.473670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.1:39501)
0811 22:28:47.474147 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:47.474277 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":238,"threads_started":1}
I20260811 22:28:47.914079 26576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:47.913313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.2:40087)
0811 22:28:47.913757 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:47.913850 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":212,"threads_started":1}
I20260811 22:28:48.344157 26578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:48.343393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.3:53609)
0811 22:28:48.343813 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:48.343921 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":209,"threads_started":1}
I20260811 22:28:48.475967 26579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:48.475259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.1:45361)
0811 22:28:48.475734 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:48.475811 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":216,"threads_started":1}
I20260811 22:28:48.915601 26580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:48.914844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.2:52331)
0811 22:28:48.915335 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:48.915424 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":234,"threads_started":1}
I20260811 22:28:49.345568 26581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:49.344901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.3:38969)
0811 22:28:49.345329 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:49.345410 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":207,"threads_started":1}
I20260811 22:28:49.477387 26582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:49.476745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.1:36301)
0811 22:28:49.477153 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:49.477229 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":210,"threads_started":1}
I20260811 22:28:49.917084 26583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:49.916371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.2:40395)
0811 22:28:49.916777 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:49.916856 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":208,"threads_started":1}
I20260811 22:28:50.347044 26584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:50.346364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.3:55761)
0811 22:28:50.346798 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:50.346879 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":240,"threads_started":1}
I20260811 22:28:50.478866 26585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:50.478176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.1:37419)
0811 22:28:50.478603 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:50.478682 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":200,"threads_started":1}
I20260811 22:28:50.918706 26588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:50.917913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.2:39069)
0811 22:28:50.918424 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:50.918512 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":290,"threads_started":1}
I20260811 22:28:51.348511 26589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:51.347843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.3:60795)
0811 22:28:51.348252 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:51.348329 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":211,"threads_started":1}
I20260811 22:28:51.480379 26594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:51.479718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.1:40837)
0811 22:28:51.480140 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:51.480220 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":216,"threads_started":1}
I20260811 22:28:51.920267 26595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:51.919578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.2:49957)
0811 22:28:51.920019 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:51.920100 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":216,"threads_started":1}
I20260811 22:28:52.350040 26597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:52.349324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.3:56203)
0811 22:28:52.349766 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:52.349852 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":222,"threads_started":1}
I20260811 22:28:52.481961 26598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:52.481222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.1:55063)
0811 22:28:52.481694 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:52.481785 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":229,"threads_started":1}
I20260811 22:28:52.921751 26602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:52.921065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.2:34087)
0811 22:28:52.921493 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:52.921584 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":221,"threads_started":1}
I20260811 22:28:53.351555 26605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:53.350815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.3:42391)
0811 22:28:53.351286 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:53.351382 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":242,"threads_started":1}
I20260811 22:28:53.483680 26607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:53.482815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.1:42681)
0811 22:28:53.483331 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:53.483421 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":193,"threads_started":1}
I20260811 22:28:53.923429 26608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:53.922629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.2:55261)
0811 22:28:53.923104 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:53.923186 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":204,"threads_started":1}
I20260811 22:28:54.353255 26609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:54.352399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.3:45011)
0811 22:28:54.352945 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:54.353028 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":254,"threads_started":1}
I20260811 22:28:54.485163 26610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:54.484507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.1:45001)
0811 22:28:54.484923 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:54.485001 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":204,"threads_started":1}
I20260811 22:28:54.924954 26611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:54.924264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.2:42023)
0811 22:28:54.924714 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:54.924794 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":222,"threads_started":1}
I20260811 22:28:55.354773 26612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:55.354117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.3:57579)
0811 22:28:55.354536 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:55.354616 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":212,"threads_started":1}
I20260811 22:28:55.486720 26614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:55.485975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.1:37817)
0811 22:28:55.486474 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:55.486552 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":242,"threads_started":1}
I20260811 22:28:55.926681 26617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:55.925855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.2:45835)
0811 22:28:55.926413 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:55.926495 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":285,"threads_started":1}
I20260811 22:28:56.356181 26618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:56.355517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.3:35065)
0811 22:28:56.355943 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:56.356020 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":195,"threads_started":1}
I20260811 22:28:56.488234 26619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:56.487527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.1:33779)
0811 22:28:56.487993 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:56.488085 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":264,"threads_started":1}
I20260811 22:28:56.928314 26620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:56.927532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.2:59421)
0811 22:28:56.928024 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:56.928137 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":230,"threads_started":1}
I20260811 22:28:57.357614 26623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:57.356962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.3:57683)
0811 22:28:57.357379 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:57.357459 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":227,"threads_started":1}
I20260811 22:28:57.489696 26624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:57.489032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.1:48701)
0811 22:28:57.489443 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:57.489528 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":204,"threads_started":1}
I20260811 22:28:57.929893 26626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:57.929216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.2:35383)
0811 22:28:57.929623 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:57.929718 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":199,"threads_started":1}
I20260811 22:28:58.359057 26627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:58.358379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.3:56795)
0811 22:28:58.358792 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:58.358882 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":207,"threads_started":1}
I20260811 22:28:58.491230 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:58.490483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.1:60993)
0811 22:28:58.490951 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:58.491032 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":276,"threads_started":1}
I20260811 22:28:58.931468 26633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:58.930756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.2:39363)
0811 22:28:58.931188 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:58.931299 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":213,"threads_started":1}
I20260811 22:28:59.360536 26636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:59.359862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.3:32895)
0811 22:28:59.360297 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:59.360376 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":241,"threads_started":1}
I20260811 22:28:59.492924 26638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:59.492026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.1:39693)
0811 22:28:59.492566 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:59.492703 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":272,"threads_started":1}
I20260811 22:28:59.933058 26639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:28:59.932289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.2:57615)
0811 22:28:59.932776 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0811 22:28:59.932865 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":253,"threads_started":1}
I20260811 22:29:00.362094 26641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:00.361301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.3:44817)
0811 22:29:00.361793 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:00.361901 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":250,"threads_started":1}
I20260811 22:29:00.494552 26644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:00.493796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.1:49573)
0811 22:29:00.494294 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:00.494371 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":274,"threads_started":1}
I20260811 22:29:00.934545 26646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:00.933840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.2:44701)
0811 22:29:00.934264 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:00.934351 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":214,"threads_started":1}
I20260811 22:29:01.363754 26648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:01.362996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.3:53025)
0811 22:29:01.363493 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:01.363574 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":269,"threads_started":1}
I20260811 22:29:01.496524 26650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:01.495681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.1:41775)
0811 22:29:01.496192 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:01.496303 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":267,"threads_started":1}
I20260811 22:29:01.936339 26651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:01.935570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.2:42155)
0811 22:29:01.936047 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:01.936159 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":218,"threads_started":1}
I20260811 22:29:02.351506 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 25773
I20260811 22:29:02.365401 26654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:02.364713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42607 (local address 127.29.106.3:39021)
0811 22:29:02.365132 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:02.365219 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":203,"threads_started":1}
I20260811 22:29:02.376770 26653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:02.352468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60180 (local address 127.29.106.1:46373)
0811 22:29:02.353036 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0811 22:29:02.353051 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0811 22:29:02.353094 (+ 43us) server_negotiation.cc:407] Connection header received
0811 22:29:02.354963 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:29:02.354973 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:29:02.355449 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:29:02.355743 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:29:02.357029 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:02.358209 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:02.359427 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:02.360100 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:02.360248 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:02.371254 (+ 11006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:29:02.371284 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:29:02.371305 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:29:02.371359 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:29:02.373928 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:29:02.374544 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:29:02.374559 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:29:02.374566 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:29:02.374724 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:29:02.374991 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:29:02.374993 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:29:02.374995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:29:02.375510 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:29:02.375811 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0811 22:29:02.376119 (+ 308us) server_negotiation.cc:299] Negotiation successful
0811 22:29:02.376488 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":241,"threads_started":1}
I20260811 22:29:02.378079 26084 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:29:02.459329 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 25773
I20260811 22:29:02.484124 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 26093
I20260811 22:29:02.501044 26659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:02.484998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54258 (local address 127.29.106.2:44241)
0811 22:29:02.485551 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0811 22:29:02.485565 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0811 22:29:02.485604 (+ 39us) server_negotiation.cc:407] Connection header received
0811 22:29:02.487298 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:29:02.487313 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:29:02.487743 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:29:02.488065 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:29:02.489491 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:02.490769 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:02.492132 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:02.492788 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:02.492953 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:02.495398 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:29:02.495406 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:29:02.495420 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:29:02.495457 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:29:02.497913 (+ 2456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:29:02.498572 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:29:02.498579 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:29:02.498585 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:29:02.498753 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:29:02.499114 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:29:02.499119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:29:02.499121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:29:02.499661 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:29:02.499958 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0811 22:29:02.500370 (+ 412us) server_negotiation.cc:299] Negotiation successful
0811 22:29:02.500770 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":257,"threads_started":1}
I20260811 22:29:02.502337 26336 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:29:02.583566 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 26093
I20260811 22:29:02.609014 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 26349
I20260811 22:29:02.625478 26664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:02.609992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38134 (local address 127.29.106.3:41165)
0811 22:29:02.610553 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0811 22:29:02.610567 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0811 22:29:02.610607 (+ 40us) server_negotiation.cc:407] Connection header received
0811 22:29:02.612253 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:29:02.612263 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:29:02.612676 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:29:02.612950 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:29:02.614223 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:02.615469 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:02.616625 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:02.617267 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:02.617394 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:02.619917 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:29:02.619924 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:29:02.619938 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:29:02.619972 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:29:02.622722 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:29:02.623396 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:29:02.623402 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:29:02.623408 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:29:02.623567 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:29:02.623864 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:29:02.623867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:29:02.623869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:29:02.624352 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:29:02.624577 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0811 22:29:02.624877 (+ 300us) server_negotiation.cc:299] Negotiation successful
0811 22:29:02.625192 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":247,"threads_started":1}
I20260811 22:29:02.626704 26562 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:29:02.709128 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 26349
2026-08-11T22:29:02Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24551 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:29:02 dist-test-slave-gjrz krb5kdc[26671](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:29:02 dist-test-slave-gjrz krb5kdc[26671](info): set up 2 sockets
Aug 11 22:29:02 dist-test-slave-gjrz krb5kdc[26671](info): commencing operation
krb5kdc: starting...
W20260811 22:29:03.978063 30120 mini_kdc.cc:121] Time spent starting KDC: real 1.223s user 0.003s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Aug 11 22:29:04 dist-test-slave-gjrz krb5kdc[26671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:29:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:29:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.106.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
I20260811 22:29:04.237984 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:45241
--webserver_interface=127.29.106.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:42673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:45241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:29:04.560608 26694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:29:04.560866 26694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:29:04.560969 26694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:29:04.567829 26694 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260811 22:29:04.567910 26694 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260811 22:29:04.567946 26694 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260811 22:29:04.567996 26694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:29:04.568048 26694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:29:04.568120 26694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:29:04.568164 26694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:29:04.568217 26694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:29:04.577678 26694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:42673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:45241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:45241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.106.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:29:04.578773 26694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:29:04.580019 26694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:29:04.591117 26703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:29:04.591230 26705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:29:04.591463 26707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:29:04.591749 26694 server_base.cc:1034] running on GCE node
I20260811 22:29:04.592442 26694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:29:04.593696 26694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:29:04.594910 26694 hybrid_clock.cc:648] HybridClock initialized: now 1786487344594846 us; error 37 us; skew 500 ppm
I20260811 22:29:04.595212 26694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:29:04 dist-test-slave-gjrz krb5kdc[26671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487344, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:29:04.598160 26694 init.cc:377] Logged in from keytab as kudu/127.29.106.62@KRBTEST.COM (short username kudu)
I20260811 22:29:04.599557 26694 webserver.cc:460] Webserver started at http://127.29.106.62:45547/ using document root <none> and password file <none>
I20260811 22:29:04.600204 26694 fs_manager.cc:359] Metadata directory not provided
I20260811 22:29:04.600301 26694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:29:04.600591 26694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:29:04.603451 26694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "1671e5e9ece446b7bd68e2a3927b30aa"
format_stamp: "Formatted at 2026-08-11 22:29:04 on dist-test-slave-gjrz"
server_key: "e0b885267508f50517aae06370a318a9"
server_key_iv: "1c95192137174e1aaf6b5a48fc8aacc3"
server_key_version: "encryptionkey@0"
I20260811 22:29:04.604223 26694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "1671e5e9ece446b7bd68e2a3927b30aa"
format_stamp: "Formatted at 2026-08-11 22:29:04 on dist-test-slave-gjrz"
server_key: "e0b885267508f50517aae06370a318a9"
server_key_iv: "1c95192137174e1aaf6b5a48fc8aacc3"
server_key_version: "encryptionkey@0"
I20260811 22:29:04.609056 26694 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260811 22:29:04.612308 26718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:29:04.613544 26694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260811 22:29:04.613678 26694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "1671e5e9ece446b7bd68e2a3927b30aa"
format_stamp: "Formatted at 2026-08-11 22:29:04 on dist-test-slave-gjrz"
server_key: "e0b885267508f50517aae06370a318a9"
server_key_iv: "1c95192137174e1aaf6b5a48fc8aacc3"
server_key_version: "encryptionkey@0"
I20260811 22:29:04.613842 26694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:29:04.625192 26694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:29:04.628755 26694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:29:04.628954 26694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:29:04.646090 26694 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.62:45241
I20260811 22:29:04.646116 26819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.62:45241 every 8 connection(s)
I20260811 22:29:04.647317 26694 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:29:04.648836 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 26694
I20260811 22:29:04.649072 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:29:04.649914 30120 external_mini_cluster.cc:1428] Setting key ca92af0c5f22df2f3d80ca495a893283
I20260811 22:29:04.651628 26820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 11 22:29:04 dist-test-slave-gjrz krb5kdc[26671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:29:04.664942 26820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa: Bootstrap starting.
I20260811 22:29:04.667944 26820 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa: Neither blocks nor log segments found. Creating new log.
I20260811 22:29:04.668923 26820 log.cc:824] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa: Log is configured to *not* fsync() on all Append() calls
I20260811 22:29:04.671587 26820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa: No bootstrap required, opened a new log
I20260811 22:29:04.672219 26830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:04.652020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56042 (local address 127.29.106.62:45241)
0811 22:29:04.652796 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0811 22:29:04.652806 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 22:29:04.652845 (+ 39us) server_negotiation.cc:407] Connection header received
0811 22:29:04.654467 (+ 1622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:29:04.654487 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:29:04.654845 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:29:04.655153 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:29:04.656923 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:04.658247 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:04.660088 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:04.660742 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:04.660873 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:04.665343 (+ 4470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:29:04.665351 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:29:04.665363 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:29:04.665408 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:29:04.668342 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:29:04.669023 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:29:04.669029 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:29:04.669035 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:29:04.669222 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:29:04.669607 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:29:04.669611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:29:04.669612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:29:04.670118 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:29:04.670397 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0811 22:29:04.670828 (+ 431us) server_negotiation.cc:299] Negotiation successful
0811 22:29:04.671203 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":383,"threads_started":1}
I20260811 22:29:04.678857 26820 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1671e5e9ece446b7bd68e2a3927b30aa" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 45241 } }
I20260811 22:29:04.679205 26820 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:29:04.679332 26820 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1671e5e9ece446b7bd68e2a3927b30aa, State: Initialized, Role: FOLLOWER
I20260811 22:29:04.679853 26820 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1671e5e9ece446b7bd68e2a3927b30aa" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 45241 } }
I20260811 22:29:04.680080 26820 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:29:04.680217 26820 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:29:04.680372 26820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:29:04.682350 26820 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1671e5e9ece446b7bd68e2a3927b30aa" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 45241 } }
I20260811 22:29:04.682808 26820 leader_election.cc:302] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1671e5e9ece446b7bd68e2a3927b30aa; no voters:
I20260811 22:29:04.683432 26820 leader_election.cc:288] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:29:04.683531 26832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:29:04.684227 26832 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [term 1 LEADER]: Becoming Leader. State: Replica: 1671e5e9ece446b7bd68e2a3927b30aa, State: Running, Role: LEADER
I20260811 22:29:04.684792 26832 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1671e5e9ece446b7bd68e2a3927b30aa" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 45241 } }
I20260811 22:29:04.685562 26820 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:29:04.687695 26836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1671e5e9ece446b7bd68e2a3927b30aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1671e5e9ece446b7bd68e2a3927b30aa" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 45241 } } }
I20260811 22:29:04.687754 26839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1671e5e9ece446b7bd68e2a3927b30aa. Latest consensus state: current_term: 1 leader_uuid: "1671e5e9ece446b7bd68e2a3927b30aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1671e5e9ece446b7bd68e2a3927b30aa" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 45241 } } }
I20260811 22:29:04.688164 26836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [sys.catalog]: This master's current role is: LEADER
I20260811 22:29:04.688169 26839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [sys.catalog]: This master's current role is: LEADER
I20260811 22:29:04.693970 26846 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:29:04.698928 26846 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:29:04.707777 26846 catalog_manager.cc:1347] Generated new cluster ID: cf441ed237fd4bdeba3ef9503c20b5c9
I20260811 22:29:04.707890 26846 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:29:04.729611 26846 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:29:04.730840 26846 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:29:04.738713 26846 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa: Generated new TSK 0
I20260811 22:29:04.739526 26846 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.106.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab.
Entry for principal kudu/127.29.106.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab.
I20260811 22:29:04.806799 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.1:0
--local_ip_for_outbound_sockets=127.29.106.1
--webserver_interface=127.29.106.1
--webserver_port=0
--tserver_master_addrs=127.29.106.62:45241
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab
--principal=kudu/127.29.106.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:42673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:29:05.134181 26870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:29:05.134431 26870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:29:05.134544 26870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:29:05.141512 26870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:29:05.141606 26870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:29:05.141721 26870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.1
I20260811 22:29:05.150887 26870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:42673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab
--principal=kudu/127.29.106.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.106.1
--webserver_port=0
--tserver_master_addrs=127.29.106.62:45241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.1
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:29:05.152014 26870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:29:05.153210 26870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:29:05.165459 26878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:29:05.165495 26880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:29:05.165653 26884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:29:05.166293 26870 server_base.cc:1034] running on GCE node
I20260811 22:29:05.166857 26870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:29:05.167752 26870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:29:05.168941 26870 hybrid_clock.cc:648] HybridClock initialized: now 1786487345168892 us; error 38 us; skew 500 ppm
I20260811 22:29:05.169243 26870 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:29:05 dist-test-slave-gjrz krb5kdc[26671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487345, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:29:05.172385 26870 init.cc:377] Logged in from keytab as kudu/127.29.106.1@KRBTEST.COM (short username kudu)
I20260811 22:29:05.173799 26870 webserver.cc:460] Webserver started at http://127.29.106.1:33193/ using document root <none> and password file <none>
I20260811 22:29:05.174413 26870 fs_manager.cc:359] Metadata directory not provided
I20260811 22:29:05.174511 26870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:29:05.174757 26870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:29:05.177549 26870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-0/data/instance:
uuid: "6f1ec800ba244930917331cd412688da"
format_stamp: "Formatted at 2026-08-11 22:29:05 on dist-test-slave-gjrz"
server_key: "95ff4a56040c2ab353ddf575e5b10852"
server_key_iv: "7c959ed106a3f169d63be6e965118b7e"
server_key_version: "encryptionkey@0"
I20260811 22:29:05.178234 26870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance:
uuid: "6f1ec800ba244930917331cd412688da"
format_stamp: "Formatted at 2026-08-11 22:29:05 on dist-test-slave-gjrz"
server_key: "95ff4a56040c2ab353ddf575e5b10852"
server_key_iv: "7c959ed106a3f169d63be6e965118b7e"
server_key_version: "encryptionkey@0"
I20260811 22:29:05.183393 26870 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260811 22:29:05.186721 26901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:29:05.188038 26870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:29:05.188182 26870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-0/wal
uuid: "6f1ec800ba244930917331cd412688da"
format_stamp: "Formatted at 2026-08-11 22:29:05 on dist-test-slave-gjrz"
server_key: "95ff4a56040c2ab353ddf575e5b10852"
server_key_iv: "7c959ed106a3f169d63be6e965118b7e"
server_key_version: "encryptionkey@0"
I20260811 22:29:05.188355 26870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:29:05.204946 26870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:29:05.208909 26870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:29:05.209178 26870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:29:05.210098 26870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:29:05.211360 26870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:29:05.211432 26870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:29:05.211551 26870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:29:05.211604 26870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:29:05.243391 26870 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.1:44961
I20260811 22:29:05.243435 27112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.1:44961 every 8 connection(s)
I20260811 22:29:05.244566 26870 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
I20260811 22:29:05.250895 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 26870
I20260811 22:29:05.251089 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance
I20260811 22:29:05.251748 30120 external_mini_cluster.cc:1428] Setting key bfd5607c2e26009979f7df5fcf9b2278
Aug 11 22:29:05 dist-test-slave-gjrz krb5kdc[26671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487345, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.1@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:29:05.264387 27116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:05.246988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:52253 (local address 127.29.106.62:45241)
0811 22:29:05.247448 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0811 22:29:05.247452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:29:05.248317 (+ 865us) server_negotiation.cc:407] Connection header received
0811 22:29:05.250128 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:29:05.250133 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:29:05.250223 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:29:05.250321 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:29:05.251899 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:05.252764 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:05.254791 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:05.255198 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:05.255364 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:05.259077 (+ 3713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:29:05.259082 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:29:05.259085 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:29:05.259117 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:29:05.261542 (+ 2425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:29:05.262330 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:29:05.262334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:29:05.262336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:29:05.262453 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:29:05.262851 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:29:05.262853 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:29:05.262855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:29:05.263094 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:29:05.263202 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0811 22:29:05.263964 (+ 762us) server_negotiation.cc:299] Negotiation successful
0811 22:29:05.264129 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":248,"threads_started":1}
I20260811 22:29:05.265590 27117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:05.247458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.1:52253)
0811 22:29:05.248160 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:05.248198 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0811 22:29:05.249862 (+ 1664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:29:05.250565 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:29:05.250573 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:29:05.251112 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:29:05.251711 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:29:05.251729 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:29:05.252948 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:29:05.252952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:29:05.254645 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:29:05.254661 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:29:05.255457 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:29:05.255465 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:29:05.255728 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:05.256377 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:29:05.256404 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:29:05.258867 (+ 2463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:29:05.261681 (+ 2814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:29:05.261690 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:29:05.261705 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:29:05.262237 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:29:05.262542 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:29:05.262545 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:29:05.262548 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:29:05.262767 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:29:05.263208 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:29:05.263215 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:29:05.263599 (+ 384us) client_negotiation.cc:769] Sending connection context
0811 22:29:05.263901 (+ 302us) client_negotiation.cc:240] Negotiation successful
0811 22:29:05.264370 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":258,"threads_started":1}
I20260811 22:29:05.267057 27114 heartbeater.cc:347] Connected to a master server at 127.29.106.62:45241
I20260811 22:29:05.267374 27114 heartbeater.cc:464] Registering TS with master...
I20260811 22:29:05.268070 27114 heartbeater.cc:511] Master 127.29.106.62:45241 requested a full tablet report, sending...
I20260811 22:29:05.270090 26744 ts_manager.cc:194] Registered new tserver with Master: 6f1ec800ba244930917331cd412688da (127.29.106.1:44961)
I20260811 22:29:05.271602 26744 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.1@KRBTEST.COM'} at 127.29.106.1:52253
WARNING: no policy specified for kudu/127.29.106.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab.
Entry for principal kudu/127.29.106.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab.
I20260811 22:29:05.311635 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.2:0
--local_ip_for_outbound_sockets=127.29.106.2
--webserver_interface=127.29.106.2
--webserver_port=0
--tserver_master_addrs=127.29.106.62:45241
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab
--principal=kudu/127.29.106.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:42673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:29:05.631512 27127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:29:05.631742 27127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:29:05.631839 27127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:29:05.638659 27127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:29:05.638751 27127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:29:05.638862 27127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.2
I20260811 22:29:05.647972 27127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:42673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab
--principal=kudu/127.29.106.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.106.2
--webserver_port=0
--tserver_master_addrs=127.29.106.62:45241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.2
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:29:05.649024 27127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:29:05.650240 27127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:29:05.662386 27136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:29:05.662431 27138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:29:05.662345 27140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:29:05.663347 27127 server_base.cc:1034] running on GCE node
I20260811 22:29:05.663902 27127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:29:05.664698 27127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:29:05.665884 27127 hybrid_clock.cc:648] HybridClock initialized: now 1786487345665840 us; error 37 us; skew 500 ppm
I20260811 22:29:05.666165 27127 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:29:05 dist-test-slave-gjrz krb5kdc[26671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487345, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:29:05.669008 27127 init.cc:377] Logged in from keytab as kudu/127.29.106.2@KRBTEST.COM (short username kudu)
I20260811 22:29:05.670327 27127 webserver.cc:460] Webserver started at http://127.29.106.2:34129/ using document root <none> and password file <none>
I20260811 22:29:05.670913 27127 fs_manager.cc:359] Metadata directory not provided
I20260811 22:29:05.671027 27127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:29:05.671343 27127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:29:05.674022 27127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-1/data/instance:
uuid: "75ab19d1756044bd858010d7be60b1d1"
format_stamp: "Formatted at 2026-08-11 22:29:05 on dist-test-slave-gjrz"
server_key: "12bdcfed3377c3b0564a823a58387d2c"
server_key_iv: "2372c0221d4b391c46b1fe1cfd630ae1"
server_key_version: "encryptionkey@0"
I20260811 22:29:05.674732 27127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance:
uuid: "75ab19d1756044bd858010d7be60b1d1"
format_stamp: "Formatted at 2026-08-11 22:29:05 on dist-test-slave-gjrz"
server_key: "12bdcfed3377c3b0564a823a58387d2c"
server_key_iv: "2372c0221d4b391c46b1fe1cfd630ae1"
server_key_version: "encryptionkey@0"
I20260811 22:29:05.679461 27127 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.007s
I20260811 22:29:05.682650 27154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:29:05.683859 27127 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:29:05.684000 27127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-1/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-1/wal
uuid: "75ab19d1756044bd858010d7be60b1d1"
format_stamp: "Formatted at 2026-08-11 22:29:05 on dist-test-slave-gjrz"
server_key: "12bdcfed3377c3b0564a823a58387d2c"
server_key_iv: "2372c0221d4b391c46b1fe1cfd630ae1"
server_key_version: "encryptionkey@0"
I20260811 22:29:05.684178 27127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:29:05.698411 27127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:29:05.702478 27127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:29:05.702744 27127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:29:05.703737 27127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:29:05.705013 27127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:29:05.705098 27127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:29:05.705219 27127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:29:05.705276 27127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:29:05.734813 27127 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.2:33035
I20260811 22:29:05.734869 27370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.2:33035 every 8 connection(s)
I20260811 22:29:05.736027 27127 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
I20260811 22:29:05.744736 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 27127
I20260811 22:29:05.744925 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance
I20260811 22:29:05.745488 30120 external_mini_cluster.cc:1428] Setting key 3897e5c7195de99a7c60a81072125706
Aug 11 22:29:05 dist-test-slave-gjrz krb5kdc[26671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487345, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.2@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:29:05.755780 27116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:05.738377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:40827 (local address 127.29.106.62:45241)
0811 22:29:05.738551 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0811 22:29:05.738557 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:29:05.739718 (+ 1161us) server_negotiation.cc:407] Connection header received
0811 22:29:05.741483 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:29:05.741488 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:29:05.741568 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:29:05.741655 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:29:05.743102 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:05.743845 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:05.745932 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:05.746443 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:05.746577 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:05.749798 (+ 3221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:29:05.749804 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:29:05.749808 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:29:05.749846 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:29:05.752717 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:29:05.753419 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:29:05.753425 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:29:05.753428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:29:05.753584 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:29:05.754024 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:29:05.754028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:29:05.754031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:29:05.754356 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:29:05.754503 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0811 22:29:05.755180 (+ 677us) server_negotiation.cc:299] Negotiation successful
0811 22:29:05.755437 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260811 22:29:05.756603 27374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:05.738821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.2:40827)
0811 22:29:05.739561 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:05.739605 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 22:29:05.741226 (+ 1621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:29:05.741898 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:29:05.741906 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:29:05.742390 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:29:05.742927 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:29:05.742944 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:29:05.744009 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:29:05.744013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:29:05.745786 (+ 1773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:29:05.745802 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:29:05.746652 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:29:05.746657 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:29:05.746848 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:05.747478 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:29:05.747507 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:29:05.749564 (+ 2057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:29:05.752872 (+ 3308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:29:05.752880 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:29:05.752892 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:29:05.753302 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:29:05.753681 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:29:05.753685 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:29:05.753689 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:29:05.753901 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:29:05.754467 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:29:05.754474 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:29:05.754815 (+ 341us) client_negotiation.cc:769] Sending connection context
0811 22:29:05.755064 (+ 249us) client_negotiation.cc:240] Negotiation successful
0811 22:29:05.755521 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":270,"threads_started":1}
I20260811 22:29:05.757951 27372 heartbeater.cc:347] Connected to a master server at 127.29.106.62:45241
I20260811 22:29:05.758266 27372 heartbeater.cc:464] Registering TS with master...
I20260811 22:29:05.759054 27372 heartbeater.cc:511] Master 127.29.106.62:45241 requested a full tablet report, sending...
I20260811 22:29:05.760610 26744 ts_manager.cc:194] Registered new tserver with Master: 75ab19d1756044bd858010d7be60b1d1 (127.29.106.2:33035)
I20260811 22:29:05.761384 26744 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.2@KRBTEST.COM'} at 127.29.106.2:40827
WARNING: no policy specified for kudu/127.29.106.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab.
Entry for principal kudu/127.29.106.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab.
I20260811 22:29:05.807250 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.3:0
--local_ip_for_outbound_sockets=127.29.106.3
--webserver_interface=127.29.106.3
--webserver_port=0
--tserver_master_addrs=127.29.106.62:45241
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab
--principal=kudu/127.29.106.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:42673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:29:06.128368 27378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:29:06.128652 27378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:29:06.128748 27378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:29:06.136024 27378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:29:06.136135 27378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:29:06.136255 27378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.3
I20260811 22:29:06.145840 27378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:42673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab
--principal=kudu/127.29.106.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.106.3
--webserver_port=0
--tserver_master_addrs=127.29.106.62:45241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.3
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:29:06.147054 27378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:29:06.148334 27378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:29:06.160933 27388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:29:06.161013 27394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:29:06.161046 27390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:29:06.161947 27378 server_base.cc:1034] running on GCE node
I20260811 22:29:06.162528 27378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:29:06.163497 27378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:29:06.164701 27378 hybrid_clock.cc:648] HybridClock initialized: now 1786487346164652 us; error 43 us; skew 500 ppm
I20260811 22:29:06.165001 27378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:29:06 dist-test-slave-gjrz krb5kdc[26671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487346, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:29:06.168074 27378 init.cc:377] Logged in from keytab as kudu/127.29.106.3@KRBTEST.COM (short username kudu)
I20260811 22:29:06.169404 27378 webserver.cc:460] Webserver started at http://127.29.106.3:36557/ using document root <none> and password file <none>
I20260811 22:29:06.170040 27378 fs_manager.cc:359] Metadata directory not provided
I20260811 22:29:06.170167 27378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:29:06.170446 27378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:29:06.173298 27378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-2/data/instance:
uuid: "75cd5e101e3b46748fc1da662c1d4c48"
format_stamp: "Formatted at 2026-08-11 22:29:06 on dist-test-slave-gjrz"
server_key: "1c1684b71b4e257e3f8bcd2b8a8278bc"
server_key_iv: "4779b690e07578b21e3c078058fd87a3"
server_key_version: "encryptionkey@0"
I20260811 22:29:06.174031 27378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance:
uuid: "75cd5e101e3b46748fc1da662c1d4c48"
format_stamp: "Formatted at 2026-08-11 22:29:06 on dist-test-slave-gjrz"
server_key: "1c1684b71b4e257e3f8bcd2b8a8278bc"
server_key_iv: "4779b690e07578b21e3c078058fd87a3"
server_key_version: "encryptionkey@0"
I20260811 22:29:06.178947 27378 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 22:29:06.182297 27413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:29:06.183558 27378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260811 22:29:06.183708 27378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-2/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-2/wal
uuid: "75cd5e101e3b46748fc1da662c1d4c48"
format_stamp: "Formatted at 2026-08-11 22:29:06 on dist-test-slave-gjrz"
server_key: "1c1684b71b4e257e3f8bcd2b8a8278bc"
server_key_iv: "4779b690e07578b21e3c078058fd87a3"
server_key_version: "encryptionkey@0"
I20260811 22:29:06.183873 27378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:29:06.201565 27378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:29:06.205602 27378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:29:06.205852 27378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:29:06.206759 27378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:29:06.208094 27378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:29:06.208168 27378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:29:06.208282 27378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:29:06.208333 27378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:29:06.238178 27378 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.3:41227
I20260811 22:29:06.238219 27631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.3:41227 every 8 connection(s)
I20260811 22:29:06.239358 27378 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
I20260811 22:29:06.241315 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 27378
I20260811 22:29:06.241472 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance
I20260811 22:29:06.242018 30120 external_mini_cluster.cc:1428] Setting key 363cae9d31640f5415a1e701a0a85296
Aug 11 22:29:06 dist-test-slave-gjrz krb5kdc[26671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487346, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.3@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:29:06.259543 27116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:06.241835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:35241 (local address 127.29.106.62:45241)
0811 22:29:06.242054 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0811 22:29:06.242061 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:29:06.243453 (+ 1392us) server_negotiation.cc:407] Connection header received
0811 22:29:06.246002 (+ 2549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:29:06.246006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:29:06.246105 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:29:06.246200 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:29:06.248281 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:06.249030 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:06.250542 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:06.250942 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:06.251049 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:06.254191 (+ 3142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:29:06.254196 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:29:06.254200 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:29:06.254230 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:29:06.256549 (+ 2319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:29:06.257206 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:29:06.257210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:29:06.257213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:29:06.257373 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:29:06.257831 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:29:06.257835 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:29:06.257838 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:29:06.258116 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:29:06.258237 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0811 22:29:06.259028 (+ 791us) server_negotiation.cc:299] Negotiation successful
0811 22:29:06.259239 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260811 22:29:06.260568 27637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:06.242380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.3:35241)
0811 22:29:06.243297 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:06.243343 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 22:29:06.245712 (+ 2369us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:29:06.246477 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:29:06.246665 (+ 188us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:29:06.247373 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:29:06.248119 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:29:06.248140 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:29:06.249192 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:29:06.249196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:29:06.250403 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:29:06.250416 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:29:06.251162 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:29:06.251170 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:29:06.251437 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:06.251989 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:29:06.252015 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:29:06.254002 (+ 1987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:29:06.256677 (+ 2675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:29:06.256685 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:29:06.256700 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:29:06.257085 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:29:06.257503 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:29:06.257507 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:29:06.257509 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:29:06.257704 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:29:06.258225 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:29:06.258232 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:29:06.258604 (+ 372us) client_negotiation.cc:769] Sending connection context
0811 22:29:06.258888 (+ 284us) client_negotiation.cc:240] Negotiation successful
0811 22:29:06.259368 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":378,"threads_started":1}
I20260811 22:29:06.261688 27633 heartbeater.cc:347] Connected to a master server at 127.29.106.62:45241
I20260811 22:29:06.261958 27633 heartbeater.cc:464] Registering TS with master...
I20260811 22:29:06.262626 27633 heartbeater.cc:511] Master 127.29.106.62:45241 requested a full tablet report, sending...
I20260811 22:29:06.263938 26744 ts_manager.cc:194] Registered new tserver with Master: 75cd5e101e3b46748fc1da662c1d4c48 (127.29.106.3:41227)
I20260811 22:29:06.264722 26744 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.3@KRBTEST.COM'} at 127.29.106.3:35241
I20260811 22:29:06.272149 30120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 22:29:06.274216 27114 heartbeater.cc:503] Master 127.29.106.62:45241 was elected leader, sending a full tablet report...
I20260811 22:29:06.288748 27116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:06.276482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56050 (local address 127.29.106.62:45241)
0811 22:29:06.276638 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0811 22:29:06.276644 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:29:06.277023 (+ 379us) server_negotiation.cc:407] Connection header received
0811 22:29:06.277190 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:29:06.277196 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:29:06.277257 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:29:06.277331 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:29:06.278783 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:06.279536 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:06.281013 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:06.281435 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:06.281539 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:06.283886 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:29:06.283892 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:29:06.283895 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:29:06.283924 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:29:06.286058 (+ 2134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:29:06.286714 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:29:06.286720 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:29:06.286723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:29:06.286861 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:29:06.287241 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:29:06.287246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:29:06.287250 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:29:06.287631 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:29:06.287860 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0811 22:29:06.288248 (+ 388us) server_negotiation.cc:299] Negotiation successful
0811 22:29:06.288454 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260811 22:29:06.292311 26744 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:29:06.298961 26744 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:29:06.321538 27662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:06.310148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44752 (local address 127.29.106.3:41227)
0811 22:29:06.313203 (+ 3055us) server_negotiation.cc:206] Beginning negotiation
0811 22:29:06.313210 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:29:06.313238 (+ 28us) server_negotiation.cc:407] Connection header received
0811 22:29:06.313317 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:29:06.313323 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:29:06.313511 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:29:06.313814 (+ 303us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:29:06.314850 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:06.316279 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:06.318937 (+ 2658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:06.320361 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:06.320496 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:06.320742 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0811 22:29:06.321024 (+ 282us) server_negotiation.cc:299] Negotiation successful
0811 22:29:06.321195 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2892,"thread_start_us":2724,"threads_started":1}
I20260811 22:29:06.321614 27666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:06.310459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:44961 (local address 127.0.0.1:55178)
0811 22:29:06.311961 (+ 1502us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:06.311978 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 22:29:06.312089 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:29:06.313626 (+ 1537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:29:06.313633 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:29:06.313664 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:29:06.314181 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:29:06.314195 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:29:06.316342 (+ 2147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:29:06.316348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:29:06.318941 (+ 2593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:29:06.318956 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:29:06.320719 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:29:06.320727 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:29:06.320877 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:06.320901 (+ 24us) client_negotiation.cc:769] Sending connection context
0811 22:29:06.320963 (+ 62us) client_negotiation.cc:240] Negotiation successful
0811 22:29:06.321211 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1378,"mutex_wait_us":356,"thread_start_us":1847,"threads_started":1}
I20260811 22:29:06.321928 27667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:06.311703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59986 (local address 127.29.106.2:33035)
0811 22:29:06.314701 (+ 2998us) server_negotiation.cc:206] Beginning negotiation
0811 22:29:06.314709 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 22:29:06.314736 (+ 27us) server_negotiation.cc:407] Connection header received
0811 22:29:06.314823 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:29:06.314830 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:29:06.315051 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:29:06.315234 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:29:06.315816 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:06.317231 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:06.319622 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:06.320593 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:06.321206 (+ 613us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:06.321396 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0811 22:29:06.321461 (+ 65us) server_negotiation.cc:299] Negotiation successful
0811 22:29:06.321625 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2834,"thread_start_us":2735,"threads_started":1}
I20260811 22:29:06.322376 27664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:06.311460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.2:33035 (local address 127.0.0.1:59986)
0811 22:29:06.312564 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:06.312581 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 22:29:06.312683 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:29:06.315239 (+ 2556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:29:06.315244 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:29:06.315316 (+ 72us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:29:06.315648 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:29:06.315660 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:29:06.317401 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:29:06.317406 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:29:06.319485 (+ 2079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:29:06.319497 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:29:06.320831 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:29:06.320837 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:29:06.320991 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:06.321020 (+ 29us) client_negotiation.cc:769] Sending connection context
0811 22:29:06.321088 (+ 68us) client_negotiation.cc:240] Negotiation successful
0811 22:29:06.322030 (+ 942us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":977,"mutex_wait_us":847,"thread_start_us":268,"threads_started":1}
I20260811 22:29:06.322572 27661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:06.309911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.3:41227 (local address 127.0.0.1:44752)
0811 22:29:06.311029 (+ 1118us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:06.311074 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 22:29:06.311229 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:29:06.313705 (+ 2476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:29:06.313709 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:29:06.313733 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:29:06.314671 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:29:06.314684 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:29:06.316400 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:29:06.316406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:29:06.318785 (+ 2379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:29:06.318797 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:29:06.320559 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:29:06.320565 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:29:06.320772 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:06.320833 (+ 61us) client_negotiation.cc:769] Sending connection context
0811 22:29:06.320964 (+ 131us) client_negotiation.cc:240] Negotiation successful
0811 22:29:06.321166 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":898,"mutex_wait_us":56,"thread_start_us":771,"threads_started":1}
I20260811 22:29:06.323376 27665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:06.310872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55178 (local address 127.29.106.1:44961)
0811 22:29:06.312968 (+ 2096us) server_negotiation.cc:206] Beginning negotiation
0811 22:29:06.312977 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 22:29:06.313006 (+ 29us) server_negotiation.cc:407] Connection header received
0811 22:29:06.313113 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:29:06.313121 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:29:06.313372 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:29:06.314134 (+ 762us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:29:06.314643 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:06.316197 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:06.319101 (+ 2904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:06.320476 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:06.322599 (+ 2123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:06.322828 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0811 22:29:06.322893 (+ 65us) server_negotiation.cc:299] Negotiation successful
0811 22:29:06.323043 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1918,"thread_start_us":1754,"threads_started":1}
I20260811 22:29:06.325992 27242 tablet_service.cc:1467] Processing CreateTablet for tablet e5b489d2b6d74fe098aa5d5ce3ae13dd (DEFAULT_TABLE table=test-table [id=2a1ebfbbac664652ae6c7e4305ba9a3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:29:06.326404 26989 tablet_service.cc:1467] Processing CreateTablet for tablet e5b489d2b6d74fe098aa5d5ce3ae13dd (DEFAULT_TABLE table=test-table [id=2a1ebfbbac664652ae6c7e4305ba9a3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:29:06.327713 27242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5b489d2b6d74fe098aa5d5ce3ae13dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:29:06.327849 26989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5b489d2b6d74fe098aa5d5ce3ae13dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:29:06.329664 27494 tablet_service.cc:1467] Processing CreateTablet for tablet e5b489d2b6d74fe098aa5d5ce3ae13dd (DEFAULT_TABLE table=test-table [id=2a1ebfbbac664652ae6c7e4305ba9a3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:29:06.330945 27494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5b489d2b6d74fe098aa5d5ce3ae13dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:29:06.342584 27669 tablet_bootstrap.cc:492] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 6f1ec800ba244930917331cd412688da: Bootstrap starting.
I20260811 22:29:06.345095 27671 tablet_bootstrap.cc:492] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75cd5e101e3b46748fc1da662c1d4c48: Bootstrap starting.
I20260811 22:29:06.346468 27670 tablet_bootstrap.cc:492] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75ab19d1756044bd858010d7be60b1d1: Bootstrap starting.
I20260811 22:29:06.346501 27669 tablet_bootstrap.cc:654] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 6f1ec800ba244930917331cd412688da: Neither blocks nor log segments found. Creating new log.
I20260811 22:29:06.347863 27669 log.cc:824] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 6f1ec800ba244930917331cd412688da: Log is configured to *not* fsync() on all Append() calls
I20260811 22:29:06.348759 27671 tablet_bootstrap.cc:654] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75cd5e101e3b46748fc1da662c1d4c48: Neither blocks nor log segments found. Creating new log.
I20260811 22:29:06.349750 27670 tablet_bootstrap.cc:654] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75ab19d1756044bd858010d7be60b1d1: Neither blocks nor log segments found. Creating new log.
I20260811 22:29:06.350067 27671 log.cc:824] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75cd5e101e3b46748fc1da662c1d4c48: Log is configured to *not* fsync() on all Append() calls
I20260811 22:29:06.351087 27670 log.cc:824] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75ab19d1756044bd858010d7be60b1d1: Log is configured to *not* fsync() on all Append() calls
I20260811 22:29:06.354554 27670 tablet_bootstrap.cc:492] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75ab19d1756044bd858010d7be60b1d1: No bootstrap required, opened a new log
I20260811 22:29:06.354643 27669 tablet_bootstrap.cc:492] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 6f1ec800ba244930917331cd412688da: No bootstrap required, opened a new log
I20260811 22:29:06.354893 27670 ts_tablet_manager.cc:1397] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75ab19d1756044bd858010d7be60b1d1: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260811 22:29:06.354960 27669 ts_tablet_manager.cc:1397] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 6f1ec800ba244930917331cd412688da: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20260811 22:29:06.354992 27671 tablet_bootstrap.cc:492] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75cd5e101e3b46748fc1da662c1d4c48: No bootstrap required, opened a new log
I20260811 22:29:06.355247 27671 ts_tablet_manager.cc:1397] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75cd5e101e3b46748fc1da662c1d4c48: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260811 22:29:06.365648 27670 raft_consensus.cc:348] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75ab19d1756044bd858010d7be60b1d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cd5e101e3b46748fc1da662c1d4c48" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41227 } } peers { permanent_uuid: "75ab19d1756044bd858010d7be60b1d1" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 33035 } } peers { permanent_uuid: "6f1ec800ba244930917331cd412688da" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 44961 } }
I20260811 22:29:06.365677 27669 raft_consensus.cc:348] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 6f1ec800ba244930917331cd412688da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cd5e101e3b46748fc1da662c1d4c48" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41227 } } peers { permanent_uuid: "75ab19d1756044bd858010d7be60b1d1" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 33035 } } peers { permanent_uuid: "6f1ec800ba244930917331cd412688da" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 44961 } }
I20260811 22:29:06.366200 27669 raft_consensus.cc:374] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 6f1ec800ba244930917331cd412688da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:29:06.366212 27670 raft_consensus.cc:374] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75ab19d1756044bd858010d7be60b1d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:29:06.366308 27670 raft_consensus.cc:729] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75ab19d1756044bd858010d7be60b1d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75ab19d1756044bd858010d7be60b1d1, State: Initialized, Role: FOLLOWER
I20260811 22:29:06.366396 27669 raft_consensus.cc:729] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 6f1ec800ba244930917331cd412688da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f1ec800ba244930917331cd412688da, State: Initialized, Role: FOLLOWER
I20260811 22:29:06.366348 27671 raft_consensus.cc:348] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75cd5e101e3b46748fc1da662c1d4c48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cd5e101e3b46748fc1da662c1d4c48" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41227 } } peers { permanent_uuid: "75ab19d1756044bd858010d7be60b1d1" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 33035 } } peers { permanent_uuid: "6f1ec800ba244930917331cd412688da" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 44961 } }
I20260811 22:29:06.366767 27671 raft_consensus.cc:374] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75cd5e101e3b46748fc1da662c1d4c48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:29:06.366915 27671 raft_consensus.cc:729] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75cd5e101e3b46748fc1da662c1d4c48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75cd5e101e3b46748fc1da662c1d4c48, State: Initialized, Role: FOLLOWER
I20260811 22:29:06.367092 27670 consensus_queue.cc:260] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75ab19d1756044bd858010d7be60b1d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cd5e101e3b46748fc1da662c1d4c48" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41227 } } peers { permanent_uuid: "75ab19d1756044bd858010d7be60b1d1" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 33035 } } peers { permanent_uuid: "6f1ec800ba244930917331cd412688da" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 44961 } }
I20260811 22:29:06.367214 27669 consensus_queue.cc:260] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 6f1ec800ba244930917331cd412688da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cd5e101e3b46748fc1da662c1d4c48" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41227 } } peers { permanent_uuid: "75ab19d1756044bd858010d7be60b1d1" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 33035 } } peers { permanent_uuid: "6f1ec800ba244930917331cd412688da" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 44961 } }
I20260811 22:29:06.367604 27671 consensus_queue.cc:260] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75cd5e101e3b46748fc1da662c1d4c48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cd5e101e3b46748fc1da662c1d4c48" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41227 } } peers { permanent_uuid: "75ab19d1756044bd858010d7be60b1d1" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 33035 } } peers { permanent_uuid: "6f1ec800ba244930917331cd412688da" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 44961 } }
I20260811 22:29:06.370113 27372 heartbeater.cc:503] Master 127.29.106.62:45241 was elected leader, sending a full tablet report...
I20260811 22:29:06.370527 27633 heartbeater.cc:503] Master 127.29.106.62:45241 was elected leader, sending a full tablet report...
I20260811 22:29:06.372537 27671 ts_tablet_manager.cc:1428] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75cd5e101e3b46748fc1da662c1d4c48: Time spent starting tablet: real 0.017s user 0.010s sys 0.003s
I20260811 22:29:06.373852 27670 ts_tablet_manager.cc:1428] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75ab19d1756044bd858010d7be60b1d1: Time spent starting tablet: real 0.019s user 0.010s sys 0.003s
I20260811 22:29:06.375672 27669 ts_tablet_manager.cc:1428] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 6f1ec800ba244930917331cd412688da: Time spent starting tablet: real 0.021s user 0.010s sys 0.008s
W20260811 22:29:06.487882 27373 tablet.cc:2369] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75ab19d1756044bd858010d7be60b1d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 22:29:06.490782 27634 tablet.cc:2369] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75cd5e101e3b46748fc1da662c1d4c48: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 22:29:06.496774 27115 tablet.cc:2369] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 6f1ec800ba244930917331cd412688da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:29:06.660872 27681 raft_consensus.cc:482] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75cd5e101e3b46748fc1da662c1d4c48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:29:06.661161 27681 raft_consensus.cc:504] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75cd5e101e3b46748fc1da662c1d4c48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cd5e101e3b46748fc1da662c1d4c48" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41227 } } peers { permanent_uuid: "75ab19d1756044bd858010d7be60b1d1" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 33035 } } peers { permanent_uuid: "6f1ec800ba244930917331cd412688da" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 44961 } }
I20260811 22:29:06.662858 27681 leader_election.cc:288] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75cd5e101e3b46748fc1da662c1d4c48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75ab19d1756044bd858010d7be60b1d1 (127.29.106.2:33035), 6f1ec800ba244930917331cd412688da (127.29.106.1:44961)
I20260811 22:29:06.668282 27667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:06.663309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:48419 (local address 127.29.106.2:33035)
0811 22:29:06.663495 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0811 22:29:06.663501 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:29:06.663521 (+ 20us) server_negotiation.cc:407] Connection header received
0811 22:29:06.663705 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:29:06.663711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:29:06.663790 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:29:06.663882 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:29:06.664276 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:06.664978 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:06.666594 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:06.667457 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:06.667598 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:06.667671 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0811 22:29:06.667924 (+ 253us) server_negotiation.cc:299] Negotiation successful
0811 22:29:06.668048 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260811 22:29:06.668282 27637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:06.663041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.2:33035 (local address 127.29.106.3:48419)
0811 22:29:06.663360 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:06.663381 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 22:29:06.663499 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:29:06.663926 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:29:06.663930 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:29:06.663955 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:29:06.664120 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:29:06.664128 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:29:06.665119 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:29:06.665123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:29:06.666429 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:29:06.666441 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:29:06.667594 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:29:06.667600 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:29:06.667775 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:06.667799 (+ 24us) client_negotiation.cc:769] Sending connection context
0811 22:29:06.667880 (+ 81us) client_negotiation.cc:240] Negotiation successful
0811 22:29:06.668013 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":38}
I20260811 22:29:06.669167 27665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:06.663317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:48125 (local address 127.29.106.1:44961)
0811 22:29:06.663533 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0811 22:29:06.663539 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:29:06.664152 (+ 613us) server_negotiation.cc:407] Connection header received
0811 22:29:06.664301 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:29:06.664305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:29:06.664363 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:29:06.664434 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:29:06.664960 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:06.665658 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:06.667340 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:06.668296 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:06.668447 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:06.668514 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0811 22:29:06.668815 (+ 301us) server_negotiation.cc:299] Negotiation successful
0811 22:29:06.668936 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260811 22:29:06.669265 27682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:06.663172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:44961 (local address 127.29.106.3:48125)
0811 22:29:06.664035 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:06.664056 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 22:29:06.664175 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:29:06.664463 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:29:06.664468 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:29:06.664491 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:29:06.664803 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:29:06.664812 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:29:06.665805 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:29:06.665808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:29:06.667158 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:29:06.667170 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:29:06.668472 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:29:06.668478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:29:06.668657 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:06.668690 (+ 33us) client_negotiation.cc:769] Sending connection context
0811 22:29:06.668771 (+ 81us) client_negotiation.cc:240] Negotiation successful
0811 22:29:06.668892 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":738,"mutex_wait_us":2,"thread_start_us":498,"threads_started":1}
I20260811 22:29:06.669449 27279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b489d2b6d74fe098aa5d5ce3ae13dd" candidate_uuid: "75cd5e101e3b46748fc1da662c1d4c48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75ab19d1756044bd858010d7be60b1d1" is_pre_election: true
I20260811 22:29:06.670047 27279 raft_consensus.cc:2454] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75ab19d1756044bd858010d7be60b1d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75cd5e101e3b46748fc1da662c1d4c48 in term 0.
I20260811 22:29:06.670225 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b489d2b6d74fe098aa5d5ce3ae13dd" candidate_uuid: "75cd5e101e3b46748fc1da662c1d4c48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f1ec800ba244930917331cd412688da" is_pre_election: true
I20260811 22:29:06.670732 27032 raft_consensus.cc:2454] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 6f1ec800ba244930917331cd412688da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75cd5e101e3b46748fc1da662c1d4c48 in term 0.
I20260811 22:29:06.670711 27418 leader_election.cc:302] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75cd5e101e3b46748fc1da662c1d4c48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75ab19d1756044bd858010d7be60b1d1, 75cd5e101e3b46748fc1da662c1d4c48; no voters:
I20260811 22:29:06.671115 27681 raft_consensus.cc:2781] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75cd5e101e3b46748fc1da662c1d4c48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:29:06.671319 27681 raft_consensus.cc:482] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75cd5e101e3b46748fc1da662c1d4c48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:29:06.671433 27681 raft_consensus.cc:3037] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75cd5e101e3b46748fc1da662c1d4c48 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:29:06.673660 27681 raft_consensus.cc:504] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75cd5e101e3b46748fc1da662c1d4c48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cd5e101e3b46748fc1da662c1d4c48" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41227 } } peers { permanent_uuid: "75ab19d1756044bd858010d7be60b1d1" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 33035 } } peers { permanent_uuid: "6f1ec800ba244930917331cd412688da" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 44961 } }
I20260811 22:29:06.674258 27681 leader_election.cc:288] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75cd5e101e3b46748fc1da662c1d4c48 [CANDIDATE]: Term 1 election: Requested vote from peers 75ab19d1756044bd858010d7be60b1d1 (127.29.106.2:33035), 6f1ec800ba244930917331cd412688da (127.29.106.1:44961)
I20260811 22:29:06.674700 27279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b489d2b6d74fe098aa5d5ce3ae13dd" candidate_uuid: "75cd5e101e3b46748fc1da662c1d4c48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75ab19d1756044bd858010d7be60b1d1"
I20260811 22:29:06.674707 27032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5b489d2b6d74fe098aa5d5ce3ae13dd" candidate_uuid: "75cd5e101e3b46748fc1da662c1d4c48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f1ec800ba244930917331cd412688da"
I20260811 22:29:06.674896 27279 raft_consensus.cc:3037] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75ab19d1756044bd858010d7be60b1d1 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:29:06.674897 27032 raft_consensus.cc:3037] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 6f1ec800ba244930917331cd412688da [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:29:06.677991 27279 raft_consensus.cc:2454] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75ab19d1756044bd858010d7be60b1d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75cd5e101e3b46748fc1da662c1d4c48 in term 1.
I20260811 22:29:06.677991 27032 raft_consensus.cc:2454] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 6f1ec800ba244930917331cd412688da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75cd5e101e3b46748fc1da662c1d4c48 in term 1.
I20260811 22:29:06.678373 27418 leader_election.cc:302] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75cd5e101e3b46748fc1da662c1d4c48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75ab19d1756044bd858010d7be60b1d1, 75cd5e101e3b46748fc1da662c1d4c48; no voters:
I20260811 22:29:06.678665 27681 raft_consensus.cc:2781] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75cd5e101e3b46748fc1da662c1d4c48 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:29:06.679124 27681 raft_consensus.cc:686] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75cd5e101e3b46748fc1da662c1d4c48 [term 1 LEADER]: Becoming Leader. State: Replica: 75cd5e101e3b46748fc1da662c1d4c48, State: Running, Role: LEADER
I20260811 22:29:06.679678 27681 consensus_queue.cc:237] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75cd5e101e3b46748fc1da662c1d4c48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cd5e101e3b46748fc1da662c1d4c48" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41227 } } peers { permanent_uuid: "75ab19d1756044bd858010d7be60b1d1" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 33035 } } peers { permanent_uuid: "6f1ec800ba244930917331cd412688da" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 44961 } }
I20260811 22:29:06.684165 26743 catalog_manager.cc:5495] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75cd5e101e3b46748fc1da662c1d4c48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 75cd5e101e3b46748fc1da662c1d4c48 (127.29.106.3). New cstate: current_term: 1 leader_uuid: "75cd5e101e3b46748fc1da662c1d4c48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cd5e101e3b46748fc1da662c1d4c48" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75ab19d1756044bd858010d7be60b1d1" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 33035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f1ec800ba244930917331cd412688da" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 44961 } health_report { overall_health: UNKNOWN } } }
I20260811 22:29:06.780382 27662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:06.775687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44766 (local address 127.29.106.3:41227)
0811 22:29:06.775847 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0811 22:29:06.775851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:29:06.775868 (+ 17us) server_negotiation.cc:407] Connection header received
0811 22:29:06.775952 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:29:06.775956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:29:06.776031 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:29:06.776150 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:29:06.776486 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:06.777201 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:06.778478 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:06.778868 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:06.778955 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:06.779412 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:29:06.779904 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:29:06.780012 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0811 22:29:06.780079 (+ 67us) server_negotiation.cc:299] Negotiation successful
0811 22:29:06.780143 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260811 22:29:06.788036 27279 raft_consensus.cc:1261] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75ab19d1756044bd858010d7be60b1d1 [term 1 FOLLOWER]: Refusing update from remote peer 75cd5e101e3b46748fc1da662c1d4c48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:29:06.788075 27032 raft_consensus.cc:1261] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 6f1ec800ba244930917331cd412688da [term 1 FOLLOWER]: Refusing update from remote peer 75cd5e101e3b46748fc1da662c1d4c48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:29:06.789037 27683 consensus_queue.cc:1035] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75cd5e101e3b46748fc1da662c1d4c48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75ab19d1756044bd858010d7be60b1d1" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 33035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:29:06.789523 27681 consensus_queue.cc:1035] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75cd5e101e3b46748fc1da662c1d4c48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f1ec800ba244930917331cd412688da" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 44961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:29:06.803647 27686 mvcc.cc:205] Tried to move back new op lower bound from 7317452172433612800 to 7317452172003504128. Current Snapshot: MvccSnapshot[applied={T|T < 7317452172433612800}]
I20260811 22:29:06.814988 26744 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:29:06.815321 26744 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:29:06.818846 26744 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa: Sending DeleteTablet for 3 replicas of tablet e5b489d2b6d74fe098aa5d5ce3ae13dd
I20260811 22:29:06.819783 27494 tablet_service.cc:1514] Processing DeleteTablet for tablet e5b489d2b6d74fe098aa5d5ce3ae13dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:29:06 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:44752
I20260811 22:29:06.819941 27242 tablet_service.cc:1514] Processing DeleteTablet for tablet e5b489d2b6d74fe098aa5d5ce3ae13dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:29:06 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:59986
I20260811 22:29:06.820127 26989 tablet_service.cc:1514] Processing DeleteTablet for tablet e5b489d2b6d74fe098aa5d5ce3ae13dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:29:06 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:55178
I20260811 22:29:06.820322 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 26870
I20260811 22:29:06.840071 27699 tablet_replica.cc:330] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75ab19d1756044bd858010d7be60b1d1: stopping tablet replica
I20260811 22:29:06.840521 27699 raft_consensus.cc:2229] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75ab19d1756044bd858010d7be60b1d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:29:06.841048 27699 raft_consensus.cc:2258] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75ab19d1756044bd858010d7be60b1d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:29:06.841862 27698 tablet_replica.cc:330] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75cd5e101e3b46748fc1da662c1d4c48: stopping tablet replica
I20260811 22:29:06.842276 27698 raft_consensus.cc:2229] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75cd5e101e3b46748fc1da662c1d4c48 [term 1 LEADER]: Raft consensus shutting down.
I20260811 22:29:06.842880 27698 raft_consensus.cc:2258] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75cd5e101e3b46748fc1da662c1d4c48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:29:06.843582 27702 tablet_replica.cc:330] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 6f1ec800ba244930917331cd412688da: stopping tablet replica
I20260811 22:29:06.843905 27702 raft_consensus.cc:2229] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 6f1ec800ba244930917331cd412688da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:29:06.844282 27702 raft_consensus.cc:2258] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 6f1ec800ba244930917331cd412688da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:29:06.846395 27702 ts_tablet_manager.cc:1905] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 6f1ec800ba244930917331cd412688da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:29:06.846971 27698 ts_tablet_manager.cc:1905] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75cd5e101e3b46748fc1da662c1d4c48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:29:06.848459 27699 ts_tablet_manager.cc:1905] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75ab19d1756044bd858010d7be60b1d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:29:06.853462 27698 ts_tablet_manager.cc:1918] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75cd5e101e3b46748fc1da662c1d4c48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:29:06.853462 27702 ts_tablet_manager.cc:1918] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 6f1ec800ba244930917331cd412688da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:29:06.853621 27702 log.cc:1196] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 6f1ec800ba244930917331cd412688da: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-0/wal/wals/e5b489d2b6d74fe098aa5d5ce3ae13dd
I20260811 22:29:06.853633 27698 log.cc:1196] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75cd5e101e3b46748fc1da662c1d4c48: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-2/wal/wals/e5b489d2b6d74fe098aa5d5ce3ae13dd
I20260811 22:29:06.854127 27698 ts_tablet_manager.cc:1939] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75cd5e101e3b46748fc1da662c1d4c48: Deleting consensus metadata
I20260811 22:29:06.854136 27702 ts_tablet_manager.cc:1939] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 6f1ec800ba244930917331cd412688da: Deleting consensus metadata
I20260811 22:29:06.855065 27699 ts_tablet_manager.cc:1918] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75ab19d1756044bd858010d7be60b1d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:29:06.855238 27699 log.cc:1196] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75ab19d1756044bd858010d7be60b1d1: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-1/wal/wals/e5b489d2b6d74fe098aa5d5ce3ae13dd
I20260811 22:29:06.855753 27699 ts_tablet_manager.cc:1939] T e5b489d2b6d74fe098aa5d5ce3ae13dd P 75ab19d1756044bd858010d7be60b1d1: Deleting consensus metadata
I20260811 22:29:06.855774 26721 catalog_manager.cc:4841] TS 6f1ec800ba244930917331cd412688da (127.29.106.1:44961): tablet e5b489d2b6d74fe098aa5d5ce3ae13dd (table test-table [id=2a1ebfbbac664652ae6c7e4305ba9a3d]) successfully deleted
I20260811 22:29:06.855769 26726 catalog_manager.cc:4841] TS 75cd5e101e3b46748fc1da662c1d4c48 (127.29.106.3:41227): tablet e5b489d2b6d74fe098aa5d5ce3ae13dd (table test-table [id=2a1ebfbbac664652ae6c7e4305ba9a3d]) successfully deleted
Aug 11 22:29:06 dist-test-slave-gjrz krb5kdc[26671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.1@KRBTEST.COM
I20260811 22:29:06.857434 26726 catalog_manager.cc:4841] TS 75ab19d1756044bd858010d7be60b1d1 (127.29.106.2:33035): tablet e5b489d2b6d74fe098aa5d5ce3ae13dd (table test-table [id=2a1ebfbbac664652ae6c7e4305ba9a3d]) successfully deleted
I20260811 22:29:06.863188 27665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:06.821568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55186 (local address 127.29.106.1:44961)
0811 22:29:06.821743 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0811 22:29:06.821750 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:29:06.845087 (+ 23337us) server_negotiation.cc:407] Connection header received
0811 22:29:06.845272 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:29:06.845276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:29:06.845337 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:29:06.845433 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:29:06.847240 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:06.848403 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:06.850482 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:06.851150 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:06.851245 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:06.857743 (+ 6498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:29:06.857755 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:29:06.857769 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:29:06.857799 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:29:06.860466 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:29:06.861134 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:29:06.861141 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:29:06.861143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:29:06.861279 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:29:06.861643 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:29:06.861647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:29:06.861649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:29:06.862009 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:29:06.862295 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0811 22:29:06.862637 (+ 342us) server_negotiation.cc:299] Negotiation successful
0811 22:29:06.862848 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260811 22:29:06.863924 27105 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:29:06.951810 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 26870
I20260811 22:29:06.990510 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 27127
Aug 11 22:29:06 dist-test-slave-gjrz krb5kdc[26671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.2@KRBTEST.COM
I20260811 22:29:07.006188 27667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:06.991255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59992 (local address 127.29.106.2:33035)
0811 22:29:06.991478 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0811 22:29:06.991484 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:29:06.991507 (+ 23us) server_negotiation.cc:407] Connection header received
0811 22:29:06.991581 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:29:06.991586 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:29:06.991669 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:29:06.991779 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:29:06.993114 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:06.994289 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:06.995726 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:06.996367 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:06.996486 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:07.000656 (+ 4170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:29:07.000672 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:29:07.000687 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:29:07.000723 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:29:07.003450 (+ 2727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:29:07.004272 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:29:07.004280 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:29:07.004282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:29:07.004422 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:29:07.004798 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:29:07.004801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:29:07.004803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:29:07.005187 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:29:07.005395 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0811 22:29:07.005735 (+ 340us) server_negotiation.cc:299] Negotiation successful
0811 22:29:07.005897 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260811 22:29:07.006830 27357 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:29:07.103296 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 27127
I20260811 22:29:07.131238 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 27378
Aug 11 22:29:07 dist-test-slave-gjrz krb5kdc[26671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.3@KRBTEST.COM
I20260811 22:29:07.148718 27662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:07.132072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44768 (local address 127.29.106.3:41227)
0811 22:29:07.132279 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0811 22:29:07.132285 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:29:07.132310 (+ 25us) server_negotiation.cc:407] Connection header received
0811 22:29:07.132432 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:29:07.132438 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:29:07.132543 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:29:07.132677 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:29:07.133970 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:07.135206 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:07.136519 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:07.137232 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:07.137363 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:07.141553 (+ 4190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:29:07.141570 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:29:07.141594 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:29:07.141650 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:29:07.145679 (+ 4029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:29:07.146401 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:29:07.146411 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:29:07.146415 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:29:07.146625 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:29:07.146993 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:29:07.146998 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:29:07.147002 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:29:07.147555 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:29:07.147909 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0811 22:29:07.148076 (+ 167us) server_negotiation.cc:299] Negotiation successful
0811 22:29:07.148306 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260811 22:29:07.149422 27619 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:29:07.271348 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 27378
I20260811 22:29:07.297001 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 26694
I20260811 22:29:07.298076 26810 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:29:07.408385 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 26694
I20260811 22:29:27.437912 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:45241
--webserver_interface=127.29.106.62
--webserver_port=45547
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:42673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:45241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:29:27.767062 27713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:29:27.767335 27713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:29:27.767477 27713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:29:27.774530 27713 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260811 22:29:27.774633 27713 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260811 22:29:27.774698 27713 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260811 22:29:27.774767 27713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:29:27.774842 27713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:29:27.774936 27713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:29:27.775000 27713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:29:27.775081 27713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:29:27.784554 27713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:42673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:45241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:45241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.106.62
--webserver_port=45547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:29:27.785681 27713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:29:27.786993 27713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:29:27.811826 27728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:29:27.811920 27723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:29:27.811913 27731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:29:27.812496 27713 server_base.cc:1034] running on GCE node
I20260811 22:29:27.813277 27713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:29:27.814705 27713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:29:27.815927 27713 hybrid_clock.cc:648] HybridClock initialized: now 1786487367815847 us; error 38 us; skew 500 ppm
I20260811 22:29:27.816298 27713 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:29:27 dist-test-slave-gjrz krb5kdc[26671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487367, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:29:27.819705 27713 init.cc:377] Logged in from keytab as kudu/127.29.106.62@KRBTEST.COM (short username kudu)
I20260811 22:29:27.821182 27713 webserver.cc:460] Webserver started at http://127.29.106.62:45547/ using document root <none> and password file <none>
I20260811 22:29:27.821913 27713 fs_manager.cc:359] Metadata directory not provided
I20260811 22:29:27.822019 27713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:29:27.827781 27713 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260811 22:29:27.830497 27744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:29:27.831797 27713 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:29:27.831957 27713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "1671e5e9ece446b7bd68e2a3927b30aa"
format_stamp: "Formatted at 2026-08-11 22:29:04 on dist-test-slave-gjrz"
server_key: "e0b885267508f50517aae06370a318a9"
server_key_iv: "1c95192137174e1aaf6b5a48fc8aacc3"
server_key_version: "encryptionkey@0"
I20260811 22:29:27.832945 27713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:29:27.857807 27713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:29:27.861781 27713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:29:27.862051 27713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:29:27.880259 27713 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.62:45241
I20260811 22:29:27.880290 27845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.62:45241 every 8 connection(s)
I20260811 22:29:27.881667 27713 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:29:27.884660 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 27713
I20260811 22:29:27.885463 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.1:44961
--local_ip_for_outbound_sockets=127.29.106.1
--tserver_master_addrs=127.29.106.62:45241
--webserver_port=33193
--webserver_interface=127.29.106.1
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab
--principal=kudu/127.29.106.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:42673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 22:29:27.888561 27846 sys_catalog.cc:263] Verifying existing consensus state
I20260811 22:29:27.891796 27846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa: Bootstrap starting.
I20260811 22:29:27.911582 27846 log.cc:824] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa: Log is configured to *not* fsync() on all Append() calls
I20260811 22:29:27.923553 27846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 22:29:27.924067 27846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa: Bootstrap complete.
I20260811 22:29:27.933041 27846 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1671e5e9ece446b7bd68e2a3927b30aa" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 45241 } }
I20260811 22:29:27.933856 27846 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1671e5e9ece446b7bd68e2a3927b30aa, State: Initialized, Role: FOLLOWER
I20260811 22:29:27.934414 27846 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1671e5e9ece446b7bd68e2a3927b30aa" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 45241 } }
I20260811 22:29:27.934653 27846 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:29:27.934765 27846 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:29:27.934927 27846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [term 1 FOLLOWER]: Advancing to term 2
I20260811 22:29:27.938814 27846 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1671e5e9ece446b7bd68e2a3927b30aa" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 45241 } }
I20260811 22:29:27.939324 27846 leader_election.cc:302] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1671e5e9ece446b7bd68e2a3927b30aa; no voters:
I20260811 22:29:27.939931 27846 leader_election.cc:288] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [CANDIDATE]: Term 2 election: Requested vote from peers
I20260811 22:29:27.940081 27857 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [term 2 FOLLOWER]: Leader election won for term 2
I20260811 22:29:27.942143 27846 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:29:27.945317 27857 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [term 2 LEADER]: Becoming Leader. State: Replica: 1671e5e9ece446b7bd68e2a3927b30aa, State: Running, Role: LEADER
I20260811 22:29:27.945984 27857 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1671e5e9ece446b7bd68e2a3927b30aa" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 45241 } }
I20260811 22:29:27.948532 27861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1671e5e9ece446b7bd68e2a3927b30aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1671e5e9ece446b7bd68e2a3927b30aa" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 45241 } } }
I20260811 22:29:27.948948 27861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [sys.catalog]: This master's current role is: LEADER
I20260811 22:29:27.949474 27862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1671e5e9ece446b7bd68e2a3927b30aa. Latest consensus state: current_term: 2 leader_uuid: "1671e5e9ece446b7bd68e2a3927b30aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1671e5e9ece446b7bd68e2a3927b30aa" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 45241 } } }
I20260811 22:29:27.949757 27862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [sys.catalog]: This master's current role is: LEADER
I20260811 22:29:27.955768 27877 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:29:27.963235 27877 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:29:27.964682 27877 catalog_manager.cc:1259] Loaded cluster ID: cf441ed237fd4bdeba3ef9503c20b5c9
I20260811 22:29:27.964824 27877 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:29:27.969111 27877 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:29:27.970651 27877 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa: Loaded TSK: 0
I20260811 22:29:28.005493 27877 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa: Generated new TSK 1
I20260811 22:29:28.005925 27877 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260811 22:29:28.237927 27851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:29:28.238188 27851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:29:28.238318 27851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:29:28.245471 27851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:29:28.245580 27851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:29:28.245724 27851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.1
I20260811 22:29:28.256629 27851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:42673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab
--principal=kudu/127.29.106.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.1:44961
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.106.1
--webserver_port=33193
--tserver_master_addrs=127.29.106.62:45241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.1
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:29:28.257869 27851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:29:28.259128 27851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:29:28.273267 27902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:29:28.273454 27903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:29:28.273927 27908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:29:28.274731 27851 server_base.cc:1034] running on GCE node
I20260811 22:29:28.275476 27851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:29:28.276602 27851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:29:28.277854 27851 hybrid_clock.cc:648] HybridClock initialized: now 1786487368277795 us; error 42 us; skew 500 ppm
I20260811 22:29:28.278228 27851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:29:28 dist-test-slave-gjrz krb5kdc[26671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487368, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:29:28.282266 27851 init.cc:377] Logged in from keytab as kudu/127.29.106.1@KRBTEST.COM (short username kudu)
I20260811 22:29:28.283883 27851 webserver.cc:460] Webserver started at http://127.29.106.1:33193/ using document root <none> and password file <none>
I20260811 22:29:28.284680 27851 fs_manager.cc:359] Metadata directory not provided
I20260811 22:29:28.284826 27851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:29:28.291347 27851 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.007s sys 0.000s
I20260811 22:29:28.294492 27925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:29:28.296021 27851 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:29:28.296198 27851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-0/wal
uuid: "6f1ec800ba244930917331cd412688da"
format_stamp: "Formatted at 2026-08-11 22:29:05 on dist-test-slave-gjrz"
server_key: "95ff4a56040c2ab353ddf575e5b10852"
server_key_iv: "7c959ed106a3f169d63be6e965118b7e"
server_key_version: "encryptionkey@0"
I20260811 22:29:28.297534 27851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:29:28.324963 27851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:29:28.330047 27851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:29:28.330313 27851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:29:28.331537 27851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:29:28.333130 27851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:29:28.333249 27851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:29:28.333355 27851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:29:28.333425 27851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:29:28.364928 27851 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.1:44961
I20260811 22:29:28.365018 28145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.1:44961 every 8 connection(s)
I20260811 22:29:28.366320 27851 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
I20260811 22:29:28.373296 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 27851
I20260811 22:29:28.374187 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.2:33035
--local_ip_for_outbound_sockets=127.29.106.2
--tserver_master_addrs=127.29.106.62:45241
--webserver_port=34129
--webserver_interface=127.29.106.2
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab
--principal=kudu/127.29.106.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:42673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 22:29:28 dist-test-slave-gjrz krb5kdc[26671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487368, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.1@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:29:28.389776 28150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:28.369227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.1:42613)
0811 22:29:28.370020 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:28.370066 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 22:29:28.371855 (+ 1789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:29:28.373142 (+ 1287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:29:28.373153 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:29:28.373878 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:29:28.374623 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:29:28.374647 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:29:28.376675 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:29:28.376680 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:29:28.378375 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:29:28.378391 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:29:28.379457 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:29:28.379465 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:29:28.379707 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:28.380338 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:29:28.380369 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:29:28.382906 (+ 2537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:29:28.386174 (+ 3268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:29:28.386180 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:29:28.386191 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:29:28.386598 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:29:28.387063 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:29:28.387066 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:29:28.387068 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:29:28.387235 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:29:28.388029 (+ 794us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:29:28.388034 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:29:28.388301 (+ 267us) client_negotiation.cc:769] Sending connection context
0811 22:29:28.388519 (+ 218us) client_negotiation.cc:240] Negotiation successful
0811 22:29:28.388856 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":319,"threads_started":1}
I20260811 22:29:28.390252 28149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:28.369017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:42613 (local address 127.29.106.62:45241)
0811 22:29:28.369623 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0811 22:29:28.369636 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 22:29:28.370182 (+ 546us) server_negotiation.cc:407] Connection header received
0811 22:29:28.372291 (+ 2109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:29:28.372304 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:29:28.372608 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:29:28.372905 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:29:28.374820 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:28.376460 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:28.378533 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:28.379202 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:28.379385 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:28.383181 (+ 3796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:29:28.383189 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:29:28.383202 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:29:28.383237 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:29:28.386033 (+ 2796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:29:28.386725 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:29:28.386733 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:29:28.386741 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:29:28.386950 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:29:28.387349 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:29:28.387353 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:29:28.387356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:29:28.387907 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:29:28.388088 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0811 22:29:28.388761 (+ 673us) server_negotiation.cc:299] Negotiation successful
0811 22:29:28.389152 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":257,"threads_started":1}
I20260811 22:29:28.393083 28147 heartbeater.cc:347] Connected to a master server at 127.29.106.62:45241
I20260811 22:29:28.393345 28147 heartbeater.cc:464] Registering TS with master...
I20260811 22:29:28.394017 28147 heartbeater.cc:511] Master 127.29.106.62:45241 requested a full tablet report, sending...
I20260811 22:29:28.396341 27780 ts_manager.cc:194] Registered new tserver with Master: 6f1ec800ba244930917331cd412688da (127.29.106.1:44961)
I20260811 22:29:28.398408 27780 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.1@KRBTEST.COM'} at 127.29.106.1:42613
W20260811 22:29:28.702453 28151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:29:28.702729 28151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:29:28.702824 28151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:29:28.710000 28151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:29:28.710112 28151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:29:28.710225 28151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.2
I20260811 22:29:28.719677 28151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:42673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab
--principal=kudu/127.29.106.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.2:33035
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.106.2
--webserver_port=34129
--tserver_master_addrs=127.29.106.62:45241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.2
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:29:28.720821 28151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:29:28.722086 28151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:29:28.734858 28164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:29:28.734905 28159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:29:28.734931 28162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:29:28.736102 28151 server_base.cc:1034] running on GCE node
I20260811 22:29:28.736681 28151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:29:28.737614 28151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:29:28.738811 28151 hybrid_clock.cc:648] HybridClock initialized: now 1786487368738767 us; error 39 us; skew 500 ppm
I20260811 22:29:28.739116 28151 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:29:28 dist-test-slave-gjrz krb5kdc[26671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487368, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:29:28.742136 28151 init.cc:377] Logged in from keytab as kudu/127.29.106.2@KRBTEST.COM (short username kudu)
I20260811 22:29:28.743592 28151 webserver.cc:460] Webserver started at http://127.29.106.2:34129/ using document root <none> and password file <none>
I20260811 22:29:28.744230 28151 fs_manager.cc:359] Metadata directory not provided
I20260811 22:29:28.744343 28151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:29:28.749437 28151 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260811 22:29:28.752058 28182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:29:28.753307 28151 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:29:28.753460 28151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-1/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-1/wal
uuid: "75ab19d1756044bd858010d7be60b1d1"
format_stamp: "Formatted at 2026-08-11 22:29:05 on dist-test-slave-gjrz"
server_key: "12bdcfed3377c3b0564a823a58387d2c"
server_key_iv: "2372c0221d4b391c46b1fe1cfd630ae1"
server_key_version: "encryptionkey@0"
I20260811 22:29:28.754382 28151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:29:28.770443 28151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:29:28.774156 28151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:29:28.774397 28151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:29:28.775364 28151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:29:28.776644 28151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:29:28.776754 28151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:29:28.776856 28151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:29:28.776929 28151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:29:28.807449 28151 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.2:33035
I20260811 22:29:28.807545 28400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.2:33035 every 8 connection(s)
I20260811 22:29:28.808795 28151 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
I20260811 22:29:28.814908 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 28151
I20260811 22:29:28.815682 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.3:41227
--local_ip_for_outbound_sockets=127.29.106.3
--tserver_master_addrs=127.29.106.62:45241
--webserver_port=36557
--webserver_interface=127.29.106.3
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab
--principal=kudu/127.29.106.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:42673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 22:29:28 dist-test-slave-gjrz krb5kdc[26671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487368, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.2@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:29:28.828667 28149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:28.811116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:59599 (local address 127.29.106.62:45241)
0811 22:29:28.811357 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0811 22:29:28.811363 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:29:28.812487 (+ 1124us) server_negotiation.cc:407] Connection header received
0811 22:29:28.814205 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:29:28.814209 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:29:28.814278 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:29:28.814363 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:29:28.816345 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:28.817039 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:28.819047 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:28.819510 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:28.819606 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:28.823431 (+ 3825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:29:28.823436 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:29:28.823439 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:29:28.823472 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:29:28.825738 (+ 2266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:29:28.826435 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:29:28.826439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:29:28.826441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:29:28.826571 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:29:28.826984 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:29:28.826987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:29:28.826989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:29:28.827257 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:29:28.827412 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0811 22:29:28.828236 (+ 824us) server_negotiation.cc:299] Negotiation successful
0811 22:29:28.828415 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260811 22:29:28.829732 28406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:28.811649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.2:59599)
0811 22:29:28.812335 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:28.812372 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0811 22:29:28.813964 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:29:28.814617 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:29:28.814628 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:29:28.815398 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:29:28.816160 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:29:28.816184 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:29:28.817208 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:29:28.817213 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:29:28.818909 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:29:28.818925 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:29:28.819738 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:29:28.819745 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:29:28.820043 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:28.820698 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:29:28.820727 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:29:28.823162 (+ 2435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:29:28.825874 (+ 2712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:29:28.825883 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:29:28.825899 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:29:28.826308 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:29:28.826675 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:29:28.826680 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:29:28.826683 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:29:28.826867 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:29:28.827412 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:29:28.827420 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:29:28.827832 (+ 412us) client_negotiation.cc:769] Sending connection context
0811 22:29:28.828126 (+ 294us) client_negotiation.cc:240] Negotiation successful
0811 22:29:28.828633 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":265,"threads_started":1}
I20260811 22:29:28.831321 28401 heartbeater.cc:347] Connected to a master server at 127.29.106.62:45241
I20260811 22:29:28.831621 28401 heartbeater.cc:464] Registering TS with master...
I20260811 22:29:28.832310 28401 heartbeater.cc:511] Master 127.29.106.62:45241 requested a full tablet report, sending...
I20260811 22:29:28.833736 27780 ts_manager.cc:194] Registered new tserver with Master: 75ab19d1756044bd858010d7be60b1d1 (127.29.106.2:33035)
I20260811 22:29:28.834568 27780 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.2@KRBTEST.COM'} at 127.29.106.2:59599
W20260811 22:29:29.148118 28410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:29:29.148391 28410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:29:29.148526 28410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:29:29.155515 28410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:29:29.155623 28410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:29:29.155774 28410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.3
I20260811 22:29:29.165139 28410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:42673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab
--principal=kudu/127.29.106.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.3:41227
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.106.3
--webserver_port=36557
--tserver_master_addrs=127.29.106.62:45241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.3
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:29:29.166311 28410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:29:29.167611 28410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:29:29.179966 28419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:29:29.180006 28421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:29:29.180042 28426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:29:29.180995 28410 server_base.cc:1034] running on GCE node
I20260811 22:29:29.181547 28410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:29:29.182413 28410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:29:29.183614 28410 hybrid_clock.cc:648] HybridClock initialized: now 1786487369183561 us; error 40 us; skew 500 ppm
I20260811 22:29:29.183902 28410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:29:29 dist-test-slave-gjrz krb5kdc[26671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487369, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:29:29.186836 28410 init.cc:377] Logged in from keytab as kudu/127.29.106.3@KRBTEST.COM (short username kudu)
I20260811 22:29:29.188184 28410 webserver.cc:460] Webserver started at http://127.29.106.3:36557/ using document root <none> and password file <none>
I20260811 22:29:29.188803 28410 fs_manager.cc:359] Metadata directory not provided
I20260811 22:29:29.188917 28410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:29:29.194885 28410 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260811 22:29:29.197463 28437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:29:29.198735 28410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 22:29:29.198882 28410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-2/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-2/wal
uuid: "75cd5e101e3b46748fc1da662c1d4c48"
format_stamp: "Formatted at 2026-08-11 22:29:06 on dist-test-slave-gjrz"
server_key: "1c1684b71b4e257e3f8bcd2b8a8278bc"
server_key_iv: "4779b690e07578b21e3c078058fd87a3"
server_key_version: "encryptionkey@0"
I20260811 22:29:29.199875 28410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:29:29.218732 28410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:29:29.222254 28410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:29:29.222457 28410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:29:29.223351 28410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:29:29.224558 28410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:29:29.224651 28410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:29:29.224735 28410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:29:29.224789 28410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:29:29.254683 28410 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.3:41227
I20260811 22:29:29.254801 28682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.3:41227 every 8 connection(s)
I20260811 22:29:29.256006 28410 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
I20260811 22:29:29.264719 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 28410
Aug 11 22:29:29 dist-test-slave-gjrz krb5kdc[26671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487369, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.3@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:29:29.276671 28149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:29.258425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:48973 (local address 127.29.106.62:45241)
0811 22:29:29.258623 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0811 22:29:29.258628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:29:29.259755 (+ 1127us) server_negotiation.cc:407] Connection header received
0811 22:29:29.261518 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:29:29.261530 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:29:29.261611 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:29:29.261698 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:29:29.263148 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:29.263912 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:29.265485 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:29.266231 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:29.266332 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:29.270341 (+ 4009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:29:29.270346 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:29:29.270349 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:29:29.270384 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:29:29.273344 (+ 2960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:29:29.274223 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:29:29.274228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:29:29.274232 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:29:29.274364 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:29:29.274913 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:29:29.274917 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:29:29.274919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:29:29.275298 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:29:29.275447 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0811 22:29:29.276078 (+ 631us) server_negotiation.cc:299] Negotiation successful
0811 22:29:29.276317 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260811 22:29:29.277479 28686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:29.258838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.3:48973)
0811 22:29:29.259612 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:29.259655 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0811 22:29:29.261254 (+ 1599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:29:29.261957 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:29:29.261965 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:29:29.262440 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:29:29.262975 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:29:29.262993 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:29:29.264072 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:29:29.264076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:29:29.265318 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:29:29.265333 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:29:29.266468 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:29:29.266476 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:29:29.266725 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:29.267392 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:29:29.267432 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:29:29.270137 (+ 2705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:29:29.273474 (+ 3337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:29:29.273482 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:29:29.273498 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:29:29.274075 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:29:29.274466 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:29:29.274470 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:29:29.274473 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:29:29.274763 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:29:29.275418 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:29:29.275424 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:29:29.275717 (+ 293us) client_negotiation.cc:769] Sending connection context
0811 22:29:29.275948 (+ 231us) client_negotiation.cc:240] Negotiation successful
0811 22:29:29.276411 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":289,"threads_started":1}
I20260811 22:29:29.279004 28684 heartbeater.cc:347] Connected to a master server at 127.29.106.62:45241
I20260811 22:29:29.279313 28684 heartbeater.cc:464] Registering TS with master...
I20260811 22:29:29.280164 28684 heartbeater.cc:511] Master 127.29.106.62:45241 requested a full tablet report, sending...
I20260811 22:29:29.280308 28691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:29.265534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44304 (local address 127.29.106.62:45241)
0811 22:29:29.266102 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0811 22:29:29.266111 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 22:29:29.266389 (+ 278us) server_negotiation.cc:407] Connection header received
0811 22:29:29.266602 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:29:29.266606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:29:29.266665 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:29:29.266757 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:29:29.268805 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:29.269766 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:29.271195 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:29.271912 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:29.272065 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:29.275115 (+ 3050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:29:29.275121 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:29:29.275125 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:29:29.275157 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:29:29.277817 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:29:29.278563 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:29:29.278568 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:29:29.278571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:29:29.278711 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:29:29.279048 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:29:29.279052 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:29:29.279055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:29:29.279364 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:29:29.279583 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0811 22:29:29.279901 (+ 318us) server_negotiation.cc:299] Negotiation successful
0811 22:29:29.280052 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":271,"threads_started":1}
I20260811 22:29:29.281839 27780 ts_manager.cc:194] Registered new tserver with Master: 75cd5e101e3b46748fc1da662c1d4c48 (127.29.106.3:41227)
I20260811 22:29:29.282814 27780 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.3@KRBTEST.COM'} at 127.29.106.3:48973
I20260811 22:29:29.282851 30120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 22:29:29.289296 28691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:29.283756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44306 (local address 127.29.106.62:45241)
0811 22:29:29.283899 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0811 22:29:29.283902 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:29:29.284354 (+ 452us) server_negotiation.cc:407] Connection header received
0811 22:29:29.284551 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:29:29.284557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:29:29.284684 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:29:29.284800 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:29:29.285229 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:29.285881 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:29.287489 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:29.287887 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:29.287992 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:29.288465 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:29:29.288702 (+ 237us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0811 22:29:29.289026 (+ 324us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44306: token expired
Metrics: {"server-negotiator.queue_time_us":40}
W20260811 22:29:29.289403 28691 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44306: token expired
W20260811 22:29:29.289873 27648 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.29.106.62:45241)
I20260811 22:29:29.301184 28691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:29.290320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44314 (local address 127.29.106.62:45241)
0811 22:29:29.290442 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0811 22:29:29.290445 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:29:29.290460 (+ 15us) server_negotiation.cc:407] Connection header received
0811 22:29:29.290574 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:29:29.290577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:29:29.290622 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:29:29.290691 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:29:29.291991 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:29.292661 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:29.293894 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:29.294288 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:29.294370 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:29.296819 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:29:29.296823 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:29:29.296826 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:29:29.296854 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:29:29.299135 (+ 2281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:29:29.299740 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:29:29.299744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:29:29.299746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:29:29.299859 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:29:29.300188 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:29:29.300191 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:29:29.300193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:29:29.300449 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:29:29.300622 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0811 22:29:29.300830 (+ 208us) server_negotiation.cc:299] Negotiation successful
0811 22:29:29.300976 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260811 22:29:29.304375 27780 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44314:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:29:29.311496 27780 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:29:29.337489 28711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:29.322000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.3:41227 (local address 127.0.0.1:34032)
0811 22:29:29.324299 (+ 2299us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:29.324362 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0811 22:29:29.324549 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:29:29.328867 (+ 4318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:29:29.328875 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:29:29.328908 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:29:29.329363 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:29:29.329380 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:29:29.332202 (+ 2822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:29:29.332209 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:29:29.334654 (+ 2445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:29:29.334670 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:29:29.336511 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:29:29.336517 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:29:29.336700 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:29.336759 (+ 59us) client_negotiation.cc:769] Sending connection context
0811 22:29:29.336869 (+ 110us) client_negotiation.cc:240] Negotiation successful
0811 22:29:29.337045 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2027,"mutex_wait_us":69,"thread_start_us":5891,"threads_started":1}
I20260811 22:29:29.338510 28709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:29.322279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34032 (local address 127.29.106.3:41227)
0811 22:29:29.328190 (+ 5911us) server_negotiation.cc:206] Beginning negotiation
0811 22:29:29.328199 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 22:29:29.328232 (+ 33us) server_negotiation.cc:407] Connection header received
0811 22:29:29.328333 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:29:29.328340 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:29:29.328608 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:29:29.329596 (+ 988us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:29:29.329876 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:29.332030 (+ 2154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:29.334820 (+ 2790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:29.336245 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:29.337739 (+ 1494us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:29.337965 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0811 22:29:29.338041 (+ 76us) server_negotiation.cc:299] Negotiation successful
0811 22:29:29.338199 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5734,"thread_start_us":5576,"threads_started":1}
I20260811 22:29:29.338949 28708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:29.322480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:44961 (local address 127.0.0.1:48416)
0811 22:29:29.328141 (+ 5661us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:29.328156 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:29:29.328358 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:29:29.329282 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:29:29.329288 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:29:29.329320 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:29:29.329736 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:29:29.329748 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:29:29.332282 (+ 2534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:29:29.332287 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:29:29.336115 (+ 3828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:29:29.336131 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:29:29.338183 (+ 2052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:29:29.338190 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:29:29.338368 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:29.338398 (+ 30us) client_negotiation.cc:769] Sending connection context
0811 22:29:29.338493 (+ 95us) client_negotiation.cc:240] Negotiation successful
0811 22:29:29.338614 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5560,"mutex_wait_us":1577,"thread_start_us":1383,"threads_started":1}
I20260811 22:29:29.339025 28712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:29.323057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48416 (local address 127.29.106.1:44961)
0811 22:29:29.327748 (+ 4691us) server_negotiation.cc:206] Beginning negotiation
0811 22:29:29.327759 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 22:29:29.328247 (+ 488us) server_negotiation.cc:407] Connection header received
0811 22:29:29.328568 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:29:29.328576 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:29:29.328791 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:29:29.328967 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:29:29.329940 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:29.332071 (+ 2131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:29.336303 (+ 4232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:29.337705 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:29.337803 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:29.337955 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0811 22:29:29.338578 (+ 623us) server_negotiation.cc:299] Negotiation successful
0811 22:29:29.338736 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4506,"thread_start_us":4389,"threads_started":1}
I20260811 22:29:29.341022 28714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:29.328289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.2:33035 (local address 127.0.0.1:33856)
0811 22:29:29.333408 (+ 5119us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:29.333427 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 22:29:29.333550 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:29:29.334402 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:29:29.334407 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:29:29.334437 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:29:29.334890 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:29:29.334900 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:29:29.336288 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:29:29.336293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:29:29.338227 (+ 1934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:29:29.338239 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:29:29.340297 (+ 2058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:29:29.340303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:29:29.340462 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:29.340492 (+ 30us) client_negotiation.cc:769] Sending connection context
0811 22:29:29.340573 (+ 81us) client_negotiation.cc:240] Negotiation successful
0811 22:29:29.340687 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4979,"mutex_wait_us":1,"thread_start_us":4789,"threads_started":1}
I20260811 22:29:29.341073 28718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:29.328704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33856 (local address 127.29.106.2:33035)
0811 22:29:29.333977 (+ 5273us) server_negotiation.cc:206] Beginning negotiation
0811 22:29:29.333983 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:29:29.334018 (+ 35us) server_negotiation.cc:407] Connection header received
0811 22:29:29.334094 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:29:29.334098 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:29:29.334236 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:29:29.334376 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:29:29.335037 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:29.336124 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:29.338379 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:29.339821 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:29.339952 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:29.340147 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0811 22:29:29.340653 (+ 506us) server_negotiation.cc:299] Negotiation successful
0811 22:29:29.340812 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5147,"thread_start_us":5056,"threads_started":1}
I20260811 22:29:29.342671 28023 tablet_service.cc:1467] Processing CreateTablet for tablet e47e7811302a47d38df2164c4d056d7b (DEFAULT_TABLE table=test-table [id=e00b1940aa924e41b4891d646ff7f6cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:29:29.342671 28553 tablet_service.cc:1467] Processing CreateTablet for tablet e47e7811302a47d38df2164c4d056d7b (DEFAULT_TABLE table=test-table [id=e00b1940aa924e41b4891d646ff7f6cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:29:29.344147 28023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e47e7811302a47d38df2164c4d056d7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:29:29.344147 28553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e47e7811302a47d38df2164c4d056d7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:29:29.344151 28272 tablet_service.cc:1467] Processing CreateTablet for tablet e47e7811302a47d38df2164c4d056d7b (DEFAULT_TABLE table=test-table [id=e00b1940aa924e41b4891d646ff7f6cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:29:29.345299 28272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e47e7811302a47d38df2164c4d056d7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:29:29.356788 28719 tablet_bootstrap.cc:492] T e47e7811302a47d38df2164c4d056d7b P 75ab19d1756044bd858010d7be60b1d1: Bootstrap starting.
I20260811 22:29:29.359163 28720 tablet_bootstrap.cc:492] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da: Bootstrap starting.
I20260811 22:29:29.360355 28719 tablet_bootstrap.cc:654] T e47e7811302a47d38df2164c4d056d7b P 75ab19d1756044bd858010d7be60b1d1: Neither blocks nor log segments found. Creating new log.
I20260811 22:29:29.360634 28723 tablet_bootstrap.cc:492] T e47e7811302a47d38df2164c4d056d7b P 75cd5e101e3b46748fc1da662c1d4c48: Bootstrap starting.
I20260811 22:29:29.362296 28719 log.cc:824] T e47e7811302a47d38df2164c4d056d7b P 75ab19d1756044bd858010d7be60b1d1: Log is configured to *not* fsync() on all Append() calls
I20260811 22:29:29.363063 28720 tablet_bootstrap.cc:654] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da: Neither blocks nor log segments found. Creating new log.
I20260811 22:29:29.363745 28723 tablet_bootstrap.cc:654] T e47e7811302a47d38df2164c4d056d7b P 75cd5e101e3b46748fc1da662c1d4c48: Neither blocks nor log segments found. Creating new log.
I20260811 22:29:29.364179 28720 log.cc:824] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da: Log is configured to *not* fsync() on all Append() calls
I20260811 22:29:29.364858 28723 log.cc:824] T e47e7811302a47d38df2164c4d056d7b P 75cd5e101e3b46748fc1da662c1d4c48: Log is configured to *not* fsync() on all Append() calls
I20260811 22:29:29.365567 28719 tablet_bootstrap.cc:492] T e47e7811302a47d38df2164c4d056d7b P 75ab19d1756044bd858010d7be60b1d1: No bootstrap required, opened a new log
I20260811 22:29:29.365900 28719 ts_tablet_manager.cc:1397] T e47e7811302a47d38df2164c4d056d7b P 75ab19d1756044bd858010d7be60b1d1: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260811 22:29:29.373395 28720 tablet_bootstrap.cc:492] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da: No bootstrap required, opened a new log
I20260811 22:29:29.373760 28720 ts_tablet_manager.cc:1397] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.007s
I20260811 22:29:29.378242 28719 raft_consensus.cc:348] T e47e7811302a47d38df2164c4d056d7b P 75ab19d1756044bd858010d7be60b1d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cd5e101e3b46748fc1da662c1d4c48" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41227 } } peers { permanent_uuid: "6f1ec800ba244930917331cd412688da" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 44961 } } peers { permanent_uuid: "75ab19d1756044bd858010d7be60b1d1" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 33035 } }
I20260811 22:29:29.379400 28719 raft_consensus.cc:374] T e47e7811302a47d38df2164c4d056d7b P 75ab19d1756044bd858010d7be60b1d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:29:29.379562 28719 raft_consensus.cc:729] T e47e7811302a47d38df2164c4d056d7b P 75ab19d1756044bd858010d7be60b1d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75ab19d1756044bd858010d7be60b1d1, State: Initialized, Role: FOLLOWER
I20260811 22:29:29.380182 28719 consensus_queue.cc:260] T e47e7811302a47d38df2164c4d056d7b P 75ab19d1756044bd858010d7be60b1d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cd5e101e3b46748fc1da662c1d4c48" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41227 } } peers { permanent_uuid: "6f1ec800ba244930917331cd412688da" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 44961 } } peers { permanent_uuid: "75ab19d1756044bd858010d7be60b1d1" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 33035 } }
I20260811 22:29:29.381063 28723 tablet_bootstrap.cc:492] T e47e7811302a47d38df2164c4d056d7b P 75cd5e101e3b46748fc1da662c1d4c48: No bootstrap required, opened a new log
I20260811 22:29:29.381315 28723 ts_tablet_manager.cc:1397] T e47e7811302a47d38df2164c4d056d7b P 75cd5e101e3b46748fc1da662c1d4c48: Time spent bootstrapping tablet: real 0.021s user 0.014s sys 0.005s
I20260811 22:29:29.383963 28401 heartbeater.cc:503] Master 127.29.106.62:45241 was elected leader, sending a full tablet report...
I20260811 22:29:29.384399 28719 ts_tablet_manager.cc:1428] T e47e7811302a47d38df2164c4d056d7b P 75ab19d1756044bd858010d7be60b1d1: Time spent starting tablet: real 0.018s user 0.012s sys 0.003s
I20260811 22:29:29.385177 28720 raft_consensus.cc:348] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cd5e101e3b46748fc1da662c1d4c48" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41227 } } peers { permanent_uuid: "6f1ec800ba244930917331cd412688da" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 44961 } } peers { permanent_uuid: "75ab19d1756044bd858010d7be60b1d1" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 33035 } }
I20260811 22:29:29.385576 28720 raft_consensus.cc:374] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:29:29.385691 28720 raft_consensus.cc:729] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f1ec800ba244930917331cd412688da, State: Initialized, Role: FOLLOWER
I20260811 22:29:29.386267 28720 consensus_queue.cc:260] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cd5e101e3b46748fc1da662c1d4c48" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41227 } } peers { permanent_uuid: "6f1ec800ba244930917331cd412688da" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 44961 } } peers { permanent_uuid: "75ab19d1756044bd858010d7be60b1d1" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 33035 } }
I20260811 22:29:29.387444 28147 heartbeater.cc:503] Master 127.29.106.62:45241 was elected leader, sending a full tablet report...
I20260811 22:29:29.387899 28720 ts_tablet_manager.cc:1428] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260811 22:29:29.390908 28723 raft_consensus.cc:348] T e47e7811302a47d38df2164c4d056d7b P 75cd5e101e3b46748fc1da662c1d4c48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cd5e101e3b46748fc1da662c1d4c48" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41227 } } peers { permanent_uuid: "6f1ec800ba244930917331cd412688da" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 44961 } } peers { permanent_uuid: "75ab19d1756044bd858010d7be60b1d1" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 33035 } }
I20260811 22:29:29.391328 28723 raft_consensus.cc:374] T e47e7811302a47d38df2164c4d056d7b P 75cd5e101e3b46748fc1da662c1d4c48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:29:29.391427 28723 raft_consensus.cc:729] T e47e7811302a47d38df2164c4d056d7b P 75cd5e101e3b46748fc1da662c1d4c48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75cd5e101e3b46748fc1da662c1d4c48, State: Initialized, Role: FOLLOWER
I20260811 22:29:29.391886 28723 consensus_queue.cc:260] T e47e7811302a47d38df2164c4d056d7b P 75cd5e101e3b46748fc1da662c1d4c48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cd5e101e3b46748fc1da662c1d4c48" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41227 } } peers { permanent_uuid: "6f1ec800ba244930917331cd412688da" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 44961 } } peers { permanent_uuid: "75ab19d1756044bd858010d7be60b1d1" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 33035 } }
I20260811 22:29:29.392736 28684 heartbeater.cc:503] Master 127.29.106.62:45241 was elected leader, sending a full tablet report...
I20260811 22:29:29.393198 28723 ts_tablet_manager.cc:1428] T e47e7811302a47d38df2164c4d056d7b P 75cd5e101e3b46748fc1da662c1d4c48: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
W20260811 22:29:29.507583 28685 tablet.cc:2369] T e47e7811302a47d38df2164c4d056d7b P 75cd5e101e3b46748fc1da662c1d4c48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:29:29.510162 28744 raft_consensus.cc:482] T e47e7811302a47d38df2164c4d056d7b P 75cd5e101e3b46748fc1da662c1d4c48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:29:29.510401 28744 raft_consensus.cc:504] T e47e7811302a47d38df2164c4d056d7b P 75cd5e101e3b46748fc1da662c1d4c48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cd5e101e3b46748fc1da662c1d4c48" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41227 } } peers { permanent_uuid: "6f1ec800ba244930917331cd412688da" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 44961 } } peers { permanent_uuid: "75ab19d1756044bd858010d7be60b1d1" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 33035 } }
I20260811 22:29:29.512081 28744 leader_election.cc:288] T e47e7811302a47d38df2164c4d056d7b P 75cd5e101e3b46748fc1da662c1d4c48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f1ec800ba244930917331cd412688da (127.29.106.1:44961), 75ab19d1756044bd858010d7be60b1d1 (127.29.106.2:33035)
I20260811 22:29:29.518249 28718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:29.512443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:45481 (local address 127.29.106.2:33035)
0811 22:29:29.512654 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0811 22:29:29.512660 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:29:29.512801 (+ 141us) server_negotiation.cc:407] Connection header received
0811 22:29:29.512985 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:29:29.512990 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:29:29.513080 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:29:29.513181 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:29:29.513679 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:29.514562 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:29.516319 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:29.517326 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:29.517478 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:29.517549 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0811 22:29:29.517872 (+ 323us) server_negotiation.cc:299] Negotiation successful
0811 22:29:29.518010 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260811 22:29:29.518261 28686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:29.512356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.2:33035 (local address 127.29.106.3:45481)
0811 22:29:29.512666 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:29.512685 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 22:29:29.512825 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:29:29.513215 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:29:29.513218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:29:29.513242 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:29:29.513503 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:29:29.513516 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:29:29.514707 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:29:29.514713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:29:29.516164 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:29:29.516177 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:29:29.517497 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:29:29.517503 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:29:29.517694 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:29.517729 (+ 35us) client_negotiation.cc:769] Sending connection context
0811 22:29:29.517828 (+ 99us) client_negotiation.cc:240] Negotiation successful
0811 22:29:29.517955 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"spinlock_wait_cycles":35712}
I20260811 22:29:29.519096 28712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:29.512559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:57863 (local address 127.29.106.1:44961)
0811 22:29:29.512748 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0811 22:29:29.512753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:29:29.513309 (+ 556us) server_negotiation.cc:407] Connection header received
0811 22:29:29.513480 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:29:29.513486 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:29:29.513562 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:29:29.513652 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:29:29.514205 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:29.515020 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:29.516942 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:29.517880 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:29.518002 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:29.518082 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0811 22:29:29.518811 (+ 729us) server_negotiation.cc:299] Negotiation successful
0811 22:29:29.518914 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260811 22:29:29.519312 28745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:29.512437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:44961 (local address 127.29.106.3:57863)
0811 22:29:29.513196 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:29.513215 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 22:29:29.513350 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:29:29.513678 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:29:29.513682 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:29:29.513706 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:29:29.514062 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:29:29.514075 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:29:29.515170 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:29:29.515174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:29:29.516786 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:29:29.516800 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:29:29.518119 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:29:29.518126 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:29:29.518300 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:29.518656 (+ 356us) client_negotiation.cc:769] Sending connection context
0811 22:29:29.518781 (+ 125us) client_negotiation.cc:240] Negotiation successful
0811 22:29:29.518931 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"mutex_wait_us":2,"thread_start_us":467,"threads_started":1}
I20260811 22:29:29.519285 28310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e7811302a47d38df2164c4d056d7b" candidate_uuid: "75cd5e101e3b46748fc1da662c1d4c48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75ab19d1756044bd858010d7be60b1d1" is_pre_election: true
I20260811 22:29:29.519791 28310 raft_consensus.cc:2454] T e47e7811302a47d38df2164c4d056d7b P 75ab19d1756044bd858010d7be60b1d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75cd5e101e3b46748fc1da662c1d4c48 in term 0.
I20260811 22:29:29.520330 28054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e7811302a47d38df2164c4d056d7b" candidate_uuid: "75cd5e101e3b46748fc1da662c1d4c48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f1ec800ba244930917331cd412688da" is_pre_election: true
I20260811 22:29:29.520465 28448 leader_election.cc:302] T e47e7811302a47d38df2164c4d056d7b P 75cd5e101e3b46748fc1da662c1d4c48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75ab19d1756044bd858010d7be60b1d1, 75cd5e101e3b46748fc1da662c1d4c48; no voters:
I20260811 22:29:29.520816 28744 raft_consensus.cc:2781] T e47e7811302a47d38df2164c4d056d7b P 75cd5e101e3b46748fc1da662c1d4c48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:29:29.520772 28054 raft_consensus.cc:2454] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75cd5e101e3b46748fc1da662c1d4c48 in term 0.
I20260811 22:29:29.520963 28744 raft_consensus.cc:482] T e47e7811302a47d38df2164c4d056d7b P 75cd5e101e3b46748fc1da662c1d4c48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:29:29.521085 28744 raft_consensus.cc:3037] T e47e7811302a47d38df2164c4d056d7b P 75cd5e101e3b46748fc1da662c1d4c48 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:29:29.523229 28744 raft_consensus.cc:504] T e47e7811302a47d38df2164c4d056d7b P 75cd5e101e3b46748fc1da662c1d4c48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cd5e101e3b46748fc1da662c1d4c48" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41227 } } peers { permanent_uuid: "6f1ec800ba244930917331cd412688da" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 44961 } } peers { permanent_uuid: "75ab19d1756044bd858010d7be60b1d1" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 33035 } }
I20260811 22:29:29.523818 28744 leader_election.cc:288] T e47e7811302a47d38df2164c4d056d7b P 75cd5e101e3b46748fc1da662c1d4c48 [CANDIDATE]: Term 1 election: Requested vote from peers 6f1ec800ba244930917331cd412688da (127.29.106.1:44961), 75ab19d1756044bd858010d7be60b1d1 (127.29.106.2:33035)
I20260811 22:29:29.524194 28054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e7811302a47d38df2164c4d056d7b" candidate_uuid: "75cd5e101e3b46748fc1da662c1d4c48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f1ec800ba244930917331cd412688da"
I20260811 22:29:29.524245 28310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47e7811302a47d38df2164c4d056d7b" candidate_uuid: "75cd5e101e3b46748fc1da662c1d4c48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75ab19d1756044bd858010d7be60b1d1"
I20260811 22:29:29.524350 28054 raft_consensus.cc:3037] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:29:29.524417 28310 raft_consensus.cc:3037] T e47e7811302a47d38df2164c4d056d7b P 75ab19d1756044bd858010d7be60b1d1 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:29:29.526468 28054 raft_consensus.cc:2454] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75cd5e101e3b46748fc1da662c1d4c48 in term 1.
I20260811 22:29:29.526577 28310 raft_consensus.cc:2454] T e47e7811302a47d38df2164c4d056d7b P 75ab19d1756044bd858010d7be60b1d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75cd5e101e3b46748fc1da662c1d4c48 in term 1.
I20260811 22:29:29.526823 28442 leader_election.cc:302] T e47e7811302a47d38df2164c4d056d7b P 75cd5e101e3b46748fc1da662c1d4c48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f1ec800ba244930917331cd412688da, 75cd5e101e3b46748fc1da662c1d4c48; no voters:
I20260811 22:29:29.527119 28744 raft_consensus.cc:2781] T e47e7811302a47d38df2164c4d056d7b P 75cd5e101e3b46748fc1da662c1d4c48 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:29:29.527525 28744 raft_consensus.cc:686] T e47e7811302a47d38df2164c4d056d7b P 75cd5e101e3b46748fc1da662c1d4c48 [term 1 LEADER]: Becoming Leader. State: Replica: 75cd5e101e3b46748fc1da662c1d4c48, State: Running, Role: LEADER
I20260811 22:29:29.528010 28744 consensus_queue.cc:237] T e47e7811302a47d38df2164c4d056d7b P 75cd5e101e3b46748fc1da662c1d4c48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cd5e101e3b46748fc1da662c1d4c48" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41227 } } peers { permanent_uuid: "6f1ec800ba244930917331cd412688da" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 44961 } } peers { permanent_uuid: "75ab19d1756044bd858010d7be60b1d1" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 33035 } }
I20260811 22:29:29.532043 27779 catalog_manager.cc:5495] T e47e7811302a47d38df2164c4d056d7b P 75cd5e101e3b46748fc1da662c1d4c48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 75cd5e101e3b46748fc1da662c1d4c48 (127.29.106.3). New cstate: current_term: 1 leader_uuid: "75cd5e101e3b46748fc1da662c1d4c48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cd5e101e3b46748fc1da662c1d4c48" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f1ec800ba244930917331cd412688da" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 44961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75ab19d1756044bd858010d7be60b1d1" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 33035 } health_report { overall_health: UNKNOWN } } }
W20260811 22:29:29.560640 28404 tablet.cc:2369] T e47e7811302a47d38df2164c4d056d7b P 75ab19d1756044bd858010d7be60b1d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:29:29.574733 28709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:29.570002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34040 (local address 127.29.106.3:41227)
0811 22:29:29.570171 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0811 22:29:29.570175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:29:29.570192 (+ 17us) server_negotiation.cc:407] Connection header received
0811 22:29:29.570321 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:29:29.570325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:29:29.570396 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:29:29.570520 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:29:29.570793 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:29.571548 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:29.572869 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:29.573286 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:29.573370 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:29.573796 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:29:29.574303 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:29:29.574414 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0811 22:29:29.574486 (+ 72us) server_negotiation.cc:299] Negotiation successful
0811 22:29:29.574546 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260811 22:29:29.582672 28310 raft_consensus.cc:1261] T e47e7811302a47d38df2164c4d056d7b P 75ab19d1756044bd858010d7be60b1d1 [term 1 FOLLOWER]: Refusing update from remote peer 75cd5e101e3b46748fc1da662c1d4c48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:29:29.582672 28054 raft_consensus.cc:1261] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [term 1 FOLLOWER]: Refusing update from remote peer 75cd5e101e3b46748fc1da662c1d4c48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:29:29.583725 28751 consensus_queue.cc:1035] T e47e7811302a47d38df2164c4d056d7b P 75cd5e101e3b46748fc1da662c1d4c48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f1ec800ba244930917331cd412688da" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 44961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:29:29.584057 28744 consensus_queue.cc:1035] T e47e7811302a47d38df2164c4d056d7b P 75cd5e101e3b46748fc1da662c1d4c48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75ab19d1756044bd858010d7be60b1d1" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 33035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:29:29.601120 28756 mvcc.cc:205] Tried to move back new op lower bound from 7317452265800257536 to 7317452265590284288. Current Snapshot: MvccSnapshot[applied={T|T < 7317452265800257536}]
I20260811 22:29:29.606287 27780 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:29:29.606525 27780 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:29:29.609881 27780 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa: Sending DeleteTablet for 3 replicas of tablet e47e7811302a47d38df2164c4d056d7b
I20260811 22:29:29.611002 28272 tablet_service.cc:1514] Processing DeleteTablet for tablet e47e7811302a47d38df2164c4d056d7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:29:29 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:33856
I20260811 22:29:29.610994 28023 tablet_service.cc:1514] Processing DeleteTablet for tablet e47e7811302a47d38df2164c4d056d7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:29:29 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:48416
I20260811 22:29:29.611097 28553 tablet_service.cc:1514] Processing DeleteTablet for tablet e47e7811302a47d38df2164c4d056d7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:29:29 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:34032
I20260811 22:29:29.612317 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 27851
I20260811 22:29:29.622184 28775 tablet_replica.cc:330] T e47e7811302a47d38df2164c4d056d7b P 75ab19d1756044bd858010d7be60b1d1: stopping tablet replica
I20260811 22:29:29.622452 28780 tablet_replica.cc:330] T e47e7811302a47d38df2164c4d056d7b P 75cd5e101e3b46748fc1da662c1d4c48: stopping tablet replica
I20260811 22:29:29.622574 28775 raft_consensus.cc:2229] T e47e7811302a47d38df2164c4d056d7b P 75ab19d1756044bd858010d7be60b1d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:29:29.622969 28780 raft_consensus.cc:2229] T e47e7811302a47d38df2164c4d056d7b P 75cd5e101e3b46748fc1da662c1d4c48 [term 1 LEADER]: Raft consensus shutting down.
I20260811 22:29:29.623030 28775 raft_consensus.cc:2258] T e47e7811302a47d38df2164c4d056d7b P 75ab19d1756044bd858010d7be60b1d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:29:29.623692 28780 raft_consensus.cc:2258] T e47e7811302a47d38df2164c4d056d7b P 75cd5e101e3b46748fc1da662c1d4c48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:29:29.625797 28780 ts_tablet_manager.cc:1905] T e47e7811302a47d38df2164c4d056d7b P 75cd5e101e3b46748fc1da662c1d4c48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:29:29.626716 28775 ts_tablet_manager.cc:1905] T e47e7811302a47d38df2164c4d056d7b P 75ab19d1756044bd858010d7be60b1d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:29:29.631744 28712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:29.613082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48430 (local address 127.29.106.1:44961)
0811 22:29:29.613651 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0811 22:29:29.613657 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:29:29.613673 (+ 16us) server_negotiation.cc:407] Connection header received
0811 22:29:29.613722 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:29:29.613728 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:29:29.613797 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:29:29.613896 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:29:29.615845 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:29.617061 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:29.619244 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:29.619930 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:29.620060 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:29.623731 (+ 3671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:29:29.623746 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:29:29.623760 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:29:29.623798 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:29:29.627621 (+ 3823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:29:29.628596 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:29:29.628605 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:29:29.628609 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:29:29.628764 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:29:29.629228 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:29:29.629232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:29:29.629235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:29:29.629693 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:29:29.629977 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0811 22:29:29.631140 (+ 1163us) server_negotiation.cc:299] Negotiation successful
0811 22:29:29.631390 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452}
I20260811 22:29:29.632380 28141 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:29:29.633512 28780 ts_tablet_manager.cc:1918] T e47e7811302a47d38df2164c4d056d7b P 75cd5e101e3b46748fc1da662c1d4c48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:29:29.633737 28780 log.cc:1196] T e47e7811302a47d38df2164c4d056d7b P 75cd5e101e3b46748fc1da662c1d4c48: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-2/wal/wals/e47e7811302a47d38df2164c4d056d7b
I20260811 22:29:29.634300 28780 ts_tablet_manager.cc:1939] T e47e7811302a47d38df2164c4d056d7b P 75cd5e101e3b46748fc1da662c1d4c48: Deleting consensus metadata
I20260811 22:29:29.634912 28775 ts_tablet_manager.cc:1918] T e47e7811302a47d38df2164c4d056d7b P 75ab19d1756044bd858010d7be60b1d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:29:29.635076 28775 log.cc:1196] T e47e7811302a47d38df2164c4d056d7b P 75ab19d1756044bd858010d7be60b1d1: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-1/wal/wals/e47e7811302a47d38df2164c4d056d7b
I20260811 22:29:29.635617 28775 ts_tablet_manager.cc:1939] T e47e7811302a47d38df2164c4d056d7b P 75ab19d1756044bd858010d7be60b1d1: Deleting consensus metadata
I20260811 22:29:29.636364 27751 catalog_manager.cc:4841] TS 75cd5e101e3b46748fc1da662c1d4c48 (127.29.106.3:41227): tablet e47e7811302a47d38df2164c4d056d7b (table test-table [id=e00b1940aa924e41b4891d646ff7f6cb]) successfully deleted
I20260811 22:29:29.637334 27751 catalog_manager.cc:4841] TS 75ab19d1756044bd858010d7be60b1d1 (127.29.106.2:33035): tablet e47e7811302a47d38df2164c4d056d7b (table test-table [id=e00b1940aa924e41b4891d646ff7f6cb]) successfully deleted
I20260811 22:29:29.724766 28779 tablet_replica.cc:330] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da: stopping tablet replica
I20260811 22:29:29.724962 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 27851
W20260811 22:29:29.760677 27749 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.106.1:44961 (error 108)
I20260811 22:29:29.760978 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 28151
I20260811 22:29:29.762779 28714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:29.762250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:44961 (local address 127.0.0.1:43686)
0811 22:29:29.762447 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:29.762552 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.1:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260811 22:29:29.763172 27749 catalog_manager.cc:4584] TS 6f1ec800ba244930917331cd412688da (127.29.106.1:44961): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e47e7811302a47d38df2164c4d056d7b: Network error: Client connection negotiation failed: client connection to 127.29.106.1:44961: connect: Connection refused (error 111)
I20260811 22:29:29.776765 28718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:29.761974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39594 (local address 127.29.106.2:33035)
0811 22:29:29.762154 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0811 22:29:29.762160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:29:29.762181 (+ 21us) server_negotiation.cc:407] Connection header received
0811 22:29:29.762369 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:29:29.762376 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:29:29.762457 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:29:29.762571 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:29:29.764244 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:29.765402 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:29.766710 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:29.767469 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:29.767594 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:29.770130 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:29:29.770150 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:29:29.770169 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:29:29.770210 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:29:29.773851 (+ 3641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:29:29.774589 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:29:29.774600 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:29:29.774604 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:29:29.774759 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:29:29.775180 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:29:29.775185 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:29:29.775188 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:29:29.775667 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:29:29.775988 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0811 22:29:29.776189 (+ 201us) server_negotiation.cc:299] Negotiation successful
0811 22:29:29.776404 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260811 22:29:29.777624 28396 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:29:29.816798 28714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:29.816286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:44961 (local address 127.0.0.1:43694)
0811 22:29:29.816508 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:29.816631 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.1:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260811 22:29:29.818830 28714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:29.818395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:44961 (local address 127.0.0.1:43708)
0811 22:29:29.818566 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:29.818663 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.1:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260811 22:29:29.866376 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 28151
I20260811 22:29:29.880062 28714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:29.879546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:44961 (local address 127.0.0.1:43714)
0811 22:29:29.879765 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:29.879871 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.1:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260811 22:29:29.881414 28714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:29.880955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:44961 (local address 127.0.0.1:43722)
0811 22:29:29.881180 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:29.881262 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.1:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260811 22:29:29.894563 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 28410
I20260811 22:29:29.909478 28709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:29.895218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55236 (local address 127.29.106.3:41227)
0811 22:29:29.895432 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0811 22:29:29.895438 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:29:29.895583 (+ 145us) server_negotiation.cc:407] Connection header received
0811 22:29:29.895762 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:29:29.895767 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:29:29.895853 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:29:29.895942 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:29:29.897260 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:29.898015 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:29.899470 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:29.899935 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:29.900056 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:29.902666 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:29:29.902679 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:29:29.902698 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:29:29.902734 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:29:29.906477 (+ 3743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:29:29.907207 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:29:29.907216 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:29:29.907220 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:29:29.907468 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:29:29.907819 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:29:29.907825 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:29:29.907828 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:29:29.908309 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:29:29.908654 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0811 22:29:29.908817 (+ 163us) server_negotiation.cc:299] Negotiation successful
0811 22:29:29.909065 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260811 22:29:29.910198 28678 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:29:29.962163 28714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:29.961654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:44961 (local address 127.0.0.1:43736)
0811 22:29:29.961868 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:29.961989 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.1:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260811 22:29:29.963192 28714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:29.962817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:44961 (local address 127.0.0.1:43748)
0811 22:29:29.963008 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:29.963082 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.1:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260811 22:29:30.031527 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 28410
I20260811 22:29:30.058339 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 27713
I20260811 22:29:30.059466 27839 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:29:30.172732 28714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:30.172069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:44961 (local address 127.0.0.1:43764)
0811 22:29:30.172269 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:30.172396 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.1:44961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260811 22:29:30.173111 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 27713
I20260811 22:29:50.202076 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:45241
--webserver_interface=127.29.106.62
--webserver_port=45547
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:42673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:45241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:29:50.529256 28789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:29:50.529531 28789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:29:50.529666 28789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:29:50.536512 28789 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260811 22:29:50.536602 28789 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260811 22:29:50.536667 28789 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260811 22:29:50.536737 28789 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:29:50.536813 28789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:29:50.536904 28789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:29:50.536966 28789 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:29:50.537040 28789 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:29:50.546217 28789 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:42673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:45241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:45241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.106.62
--webserver_port=45547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:29:50.547328 28789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:29:50.548516 28789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:29:50.560047 28810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:29:50.560108 28803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:29:50.560086 28806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:29:50.560258 28789 server_base.cc:1034] running on GCE node
I20260811 22:29:50.561360 28789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:29:50.562652 28789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:29:50.563860 28789 hybrid_clock.cc:648] HybridClock initialized: now 1786487390563793 us; error 36 us; skew 500 ppm
I20260811 22:29:50.564181 28789 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:29:50 dist-test-slave-gjrz krb5kdc[26671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487390, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:29:50.567202 28789 init.cc:377] Logged in from keytab as kudu/127.29.106.62@KRBTEST.COM (short username kudu)
I20260811 22:29:50.568488 28789 webserver.cc:460] Webserver started at http://127.29.106.62:45547/ using document root <none> and password file <none>
I20260811 22:29:50.569129 28789 fs_manager.cc:359] Metadata directory not provided
I20260811 22:29:50.569228 28789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:29:50.574419 28789 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.001s
I20260811 22:29:50.577275 28820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:29:50.578485 28789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 22:29:50.578648 28789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "1671e5e9ece446b7bd68e2a3927b30aa"
format_stamp: "Formatted at 2026-08-11 22:29:04 on dist-test-slave-gjrz"
server_key: "e0b885267508f50517aae06370a318a9"
server_key_iv: "1c95192137174e1aaf6b5a48fc8aacc3"
server_key_version: "encryptionkey@0"
I20260811 22:29:50.579633 28789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:29:50.613384 28789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:29:50.617304 28789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:29:50.617568 28789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:29:50.634459 28789 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.62:45241
I20260811 22:29:50.634502 28914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.62:45241 every 8 connection(s)
I20260811 22:29:50.635792 28789 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:29:50.639562 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 28789
I20260811 22:29:50.640592 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.1:44961
--local_ip_for_outbound_sockets=127.29.106.1
--tserver_master_addrs=127.29.106.62:45241
--webserver_port=33193
--webserver_interface=127.29.106.1
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab
--principal=kudu/127.29.106.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:42673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 22:29:50.643658 28916 sys_catalog.cc:263] Verifying existing consensus state
I20260811 22:29:50.647790 28916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa: Bootstrap starting.
I20260811 22:29:50.669752 28916 log.cc:824] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa: Log is configured to *not* fsync() on all Append() calls
I20260811 22:29:50.686499 28916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 22:29:50.687002 28916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa: Bootstrap complete.
I20260811 22:29:50.696054 28916 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1671e5e9ece446b7bd68e2a3927b30aa" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 45241 } }
I20260811 22:29:50.696924 28916 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1671e5e9ece446b7bd68e2a3927b30aa, State: Initialized, Role: FOLLOWER
I20260811 22:29:50.697541 28916 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1671e5e9ece446b7bd68e2a3927b30aa" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 45241 } }
I20260811 22:29:50.697788 28916 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:29:50.697894 28916 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:29:50.698110 28916 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [term 2 FOLLOWER]: Advancing to term 3
I20260811 22:29:50.701894 28916 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1671e5e9ece446b7bd68e2a3927b30aa" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 45241 } }
I20260811 22:29:50.702359 28916 leader_election.cc:302] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1671e5e9ece446b7bd68e2a3927b30aa; no voters:
I20260811 22:29:50.703014 28916 leader_election.cc:288] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [CANDIDATE]: Term 3 election: Requested vote from peers
I20260811 22:29:50.703168 28926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [term 3 FOLLOWER]: Leader election won for term 3
I20260811 22:29:50.704308 28926 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [term 3 LEADER]: Becoming Leader. State: Replica: 1671e5e9ece446b7bd68e2a3927b30aa, State: Running, Role: LEADER
I20260811 22:29:50.704921 28926 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1671e5e9ece446b7bd68e2a3927b30aa" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 45241 } }
I20260811 22:29:50.705178 28916 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:29:50.707423 28932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1671e5e9ece446b7bd68e2a3927b30aa. Latest consensus state: current_term: 3 leader_uuid: "1671e5e9ece446b7bd68e2a3927b30aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1671e5e9ece446b7bd68e2a3927b30aa" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 45241 } } }
I20260811 22:29:50.707819 28932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [sys.catalog]: This master's current role is: LEADER
I20260811 22:29:50.708382 28931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "1671e5e9ece446b7bd68e2a3927b30aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1671e5e9ece446b7bd68e2a3927b30aa" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 45241 } } }
I20260811 22:29:50.708665 28931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa [sys.catalog]: This master's current role is: LEADER
I20260811 22:29:50.716420 28940 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:29:50.725994 28940 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:29:50.727533 28940 catalog_manager.cc:1259] Loaded cluster ID: cf441ed237fd4bdeba3ef9503c20b5c9
I20260811 22:29:50.727679 28940 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:29:50.731721 28940 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:29:50.733507 28940 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa: Loaded TSK: 0
I20260811 22:29:50.733713 28940 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1671e5e9ece446b7bd68e2a3927b30aa: Loaded TSK: 1
F20260811 22:29:50.734867 28940 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f1b3cbbe9ad google::LogMessage::Fail() at ??:0
@ 0x7f1b3cbc188a google::LogMessage::SendToLog() at ??:0
@ 0x7f1b3cbbe4de google::LogMessage::Flush() at ??:0
@ 0x7f1b3cbbfa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f1b409227d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f1b4091173e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f1b525ed1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f1b525ec783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f1b526782a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f1b526780c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f1b526c25e9 std::function<>::operator()() at ??:0
@ 0x7f1b525f0cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f1b525eec36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f1b52676610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f1b52676439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f1b3f80a345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1b3f80d91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f1b3f80d799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f1b3f7e8eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1b42ac26db start_thread at ??:0
@ 0x7f1b3ba5861f clone at ??:0
W20260811 22:29:51.014962 28919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:29:51.015234 28919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:29:51.015484 28919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:29:51.022612 28919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:29:51.022720 28919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:29:51.022871 28919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.1
I20260811 22:29:51.032291 28919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:42673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab
--principal=kudu/127.29.106.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.1:44961
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.106.1
--webserver_port=33193
--tserver_master_addrs=127.29.106.62:45241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.1
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:29:51.033437 28919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:29:51.034675 28919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:29:51.047369 29023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:29:51.047411 29024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:29:51.047436 29028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:29:51.048259 28919 server_base.cc:1034] running on GCE node
I20260811 22:29:51.048861 28919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:29:51.049790 28919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:29:51.050987 28919 hybrid_clock.cc:648] HybridClock initialized: now 1786487391050929 us; error 47 us; skew 500 ppm
I20260811 22:29:51.051311 28919 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:29:51 dist-test-slave-gjrz krb5kdc[26671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487391, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:29:51.054339 28919 init.cc:377] Logged in from keytab as kudu/127.29.106.1@KRBTEST.COM (short username kudu)
I20260811 22:29:51.055689 28919 webserver.cc:460] Webserver started at http://127.29.106.1:33193/ using document root <none> and password file <none>
I20260811 22:29:51.056313 28919 fs_manager.cc:359] Metadata directory not provided
I20260811 22:29:51.056433 28919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:29:51.061528 28919 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260811 22:29:51.064303 29047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:29:51.065533 28919 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260811 22:29:51.065688 28919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-0/wal
uuid: "6f1ec800ba244930917331cd412688da"
format_stamp: "Formatted at 2026-08-11 22:29:05 on dist-test-slave-gjrz"
server_key: "95ff4a56040c2ab353ddf575e5b10852"
server_key_iv: "7c959ed106a3f169d63be6e965118b7e"
server_key_version: "encryptionkey@0"
I20260811 22:29:51.066615 28919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:29:51.080123 28919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:29:51.083983 28919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:29:51.084245 28919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:29:51.085177 28919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:29:51.086910 29060 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260811 22:29:51.092406 28919 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260811 22:29:51.092530 28919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260811 22:29:51.092641 28919 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260811 22:29:51.095710 28919 ts_tablet_manager.cc:610] Registered 1 tablets
I20260811 22:29:51.095789 28919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20260811 22:29:51.096119 29060 tablet_bootstrap.cc:492] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da: Bootstrap starting.
I20260811 22:29:51.122802 29060 log.cc:824] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da: Log is configured to *not* fsync() on all Append() calls
I20260811 22:29:51.132376 28919 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.1:44961
I20260811 22:29:51.132421 29294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.1:44961 every 8 connection(s)
I20260811 22:29:51.133561 28919 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
I20260811 22:29:51.135823 29060 tablet_bootstrap.cc:492] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 22:29:51.136343 29060 tablet_bootstrap.cc:492] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da: Bootstrap complete.
I20260811 22:29:51.137988 29060 ts_tablet_manager.cc:1397] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da: Time spent bootstrapping tablet: real 0.042s user 0.035s sys 0.004s
I20260811 22:29:51.138742 29299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:51.136425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.1:49271)
0811 22:29:51.137277 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:51.137567 (+ 290us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":344,"threads_started":1}
W20260811 22:29:51.139472 29296 heartbeater.cc:644] Failed to heartbeat to 127.29.106.62:45241 (0 consecutive failures): Network error: Failed to ping master at 127.29.106.62:45241: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
I20260811 22:29:51.140431 29299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:51.140019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.1:49727)
0811 22:29:51.140194 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:51.140286 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260811 22:29:51.141448 29299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:51.141037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.1:33581)
0811 22:29:51.141199 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:51.141297 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260811 22:29:51.141608 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 28919
W20260811 22:29:51.141741 29296 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260811 22:29:51.142408 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.2:33035
--local_ip_for_outbound_sockets=127.29.106.2
--tserver_master_addrs=127.29.106.62:45241
--webserver_port=34129
--webserver_interface=127.29.106.2
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab
--principal=kudu/127.29.106.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:42673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 22:29:51.146837 29060 raft_consensus.cc:348] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cd5e101e3b46748fc1da662c1d4c48" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41227 } } peers { permanent_uuid: "6f1ec800ba244930917331cd412688da" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 44961 } } peers { permanent_uuid: "75ab19d1756044bd858010d7be60b1d1" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 33035 } }
I20260811 22:29:51.147991 29060 raft_consensus.cc:729] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f1ec800ba244930917331cd412688da, State: Initialized, Role: FOLLOWER
I20260811 22:29:51.148658 29060 consensus_queue.cc:260] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cd5e101e3b46748fc1da662c1d4c48" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41227 } } peers { permanent_uuid: "6f1ec800ba244930917331cd412688da" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 44961 } } peers { permanent_uuid: "75ab19d1756044bd858010d7be60b1d1" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 33035 } }
I20260811 22:29:51.150293 29299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:51.149870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.1:35091)
0811 22:29:51.150099 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:51.150174 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260811 22:29:51.150336 29060 ts_tablet_manager.cc:1428] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da: Time spent starting tablet: real 0.012s user 0.014s sys 0.000s
I20260811 22:29:51.151254 29299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:51.150905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.1:58441)
0811 22:29:51.151045 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:51.151121 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260811 22:29:51.473377 29308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:29:51.473640 29308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:29:51.473779 29308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:29:51.480780 29308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:29:51.480886 29308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:29:51.481040 29308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.2
I20260811 22:29:51.490186 29308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:42673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab
--principal=kudu/127.29.106.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.2:33035
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.106.2
--webserver_port=34129
--tserver_master_addrs=127.29.106.62:45241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.2
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:29:51.491539 29308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:29:51.493141 29308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:29:51.506091 29327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:29:51.506170 29321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:29:51.506137 29319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:29:51.507088 29308 server_base.cc:1034] running on GCE node
I20260811 22:29:51.507692 29308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:29:51.508524 29308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:29:51.509724 29308 hybrid_clock.cc:648] HybridClock initialized: now 1786487391509682 us; error 35 us; skew 500 ppm
I20260811 22:29:51.509982 29308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:29:51 dist-test-slave-gjrz krb5kdc[26671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487391, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:29:51.512854 29308 init.cc:377] Logged in from keytab as kudu/127.29.106.2@KRBTEST.COM (short username kudu)
I20260811 22:29:51.514109 29308 webserver.cc:460] Webserver started at http://127.29.106.2:34129/ using document root <none> and password file <none>
I20260811 22:29:51.514703 29308 fs_manager.cc:359] Metadata directory not provided
I20260811 22:29:51.514801 29308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:29:51.519703 29308 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:29:51.522169 29340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:29:51.523394 29308 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:29:51.523517 29308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-1/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-1/wal
uuid: "75ab19d1756044bd858010d7be60b1d1"
format_stamp: "Formatted at 2026-08-11 22:29:05 on dist-test-slave-gjrz"
server_key: "12bdcfed3377c3b0564a823a58387d2c"
server_key_iv: "2372c0221d4b391c46b1fe1cfd630ae1"
server_key_version: "encryptionkey@0"
I20260811 22:29:51.524415 29308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:29:51.546123 29308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:29:51.549556 29308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:29:51.549798 29308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:29:51.550647 29308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:29:51.551872 29308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:29:51.551954 29308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:29:51.552031 29308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:29:51.552091 29308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:29:51.581362 29308 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.2:33035
I20260811 22:29:51.581425 29570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.2:33035 every 8 connection(s)
I20260811 22:29:51.582674 29308 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
I20260811 22:29:51.587036 29575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:51.585323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.2:50167)
0811 22:29:51.586005 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:51.586248 (+ 243us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":263,"threads_started":1}
W20260811 22:29:51.587672 29571 heartbeater.cc:644] Failed to heartbeat to 127.29.106.62:45241 (0 consecutive failures): Network error: Failed to ping master at 127.29.106.62:45241: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
I20260811 22:29:51.588357 29575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:51.588068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.2:52627)
0811 22:29:51.588196 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:51.588259 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260811 22:29:51.589072 29575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:51.588808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.2:37189)
0811 22:29:51.588904 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:51.588972 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260811 22:29:51.589289 29571 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260811 22:29:51.590481 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 29308
I20260811 22:29:51.591066 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.3:41227
--local_ip_for_outbound_sockets=127.29.106.3
--tserver_master_addrs=127.29.106.62:45241
--webserver_port=36557
--webserver_interface=127.29.106.3
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab
--principal=kudu/127.29.106.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:42673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:29:51.917500 29576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:29:51.917740 29576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:29:51.917869 29576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:29:51.924870 29576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:29:51.924971 29576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:29:51.925112 29576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.3
I20260811 22:29:51.934315 29576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:42673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab
--principal=kudu/127.29.106.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.3:41227
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.106.3
--webserver_port=36557
--tserver_master_addrs=127.29.106.62:45241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.3
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:29:51.935456 29576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:29:51.936684 29576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:29:51.948899 29595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:29:51.948868 29597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:29:51.948987 29592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:29:51.949648 29576 server_base.cc:1034] running on GCE node
I20260811 22:29:51.950220 29576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:29:51.951092 29576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:29:51.952332 29576 hybrid_clock.cc:648] HybridClock initialized: now 1786487391952273 us; error 40 us; skew 500 ppm
I20260811 22:29:51.952641 29576 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:29:51 dist-test-slave-gjrz krb5kdc[26671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487391, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:29:51.959885 29576 init.cc:377] Logged in from keytab as kudu/127.29.106.3@KRBTEST.COM (short username kudu)
I20260811 22:29:51.961249 29576 webserver.cc:460] Webserver started at http://127.29.106.3:36557/ using document root <none> and password file <none>
I20260811 22:29:51.961900 29576 fs_manager.cc:359] Metadata directory not provided
I20260811 22:29:51.962034 29576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:29:51.967182 29576 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260811 22:29:51.969791 29615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:29:51.971094 29576 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 22:29:51.971237 29576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-2/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-2/wal
uuid: "75cd5e101e3b46748fc1da662c1d4c48"
format_stamp: "Formatted at 2026-08-11 22:29:06 on dist-test-slave-gjrz"
server_key: "1c1684b71b4e257e3f8bcd2b8a8278bc"
server_key_iv: "4779b690e07578b21e3c078058fd87a3"
server_key_version: "encryptionkey@0"
I20260811 22:29:51.972218 29576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:29:51.988348 29576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:29:51.992317 29576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:29:51.992570 29576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:29:51.993468 29576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:29:51.994701 29576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:29:51.994812 29576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:29:51.994920 29576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:29:51.994995 29576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:29:52.027577 29576 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.3:41227
I20260811 22:29:52.027689 29837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.3:41227 every 8 connection(s)
I20260811 22:29:52.029172 29576 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
I20260811 22:29:52.033842 29844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:52.032080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.3:44065)
0811 22:29:52.032787 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:52.033013 (+ 226us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":269,"threads_started":1}
W20260811 22:29:52.034493 29840 heartbeater.cc:644] Failed to heartbeat to 127.29.106.62:45241 (0 consecutive failures): Network error: Failed to ping master at 127.29.106.62:45241: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
I20260811 22:29:52.035250 29844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:52.034940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.3:47303)
0811 22:29:52.035075 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:52.035151 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260811 22:29:52.036026 29844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:52.035741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.3:49465)
0811 22:29:52.035857 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:52.035927 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260811 22:29:52.036263 29840 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260811 22:29:52.036995 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 29576
I20260811 22:29:52.152776 29847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:52.152059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.1:50857)
0811 22:29:52.152479 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:52.152581 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":195,"threads_started":1}
I20260811 22:29:52.590466 29848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:52.589730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.2:34805)
0811 22:29:52.590165 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:52.590249 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":211,"threads_started":1}
I20260811 22:29:52.739360 29849 raft_consensus.cc:482] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:29:52.739634 29849 raft_consensus.cc:504] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cd5e101e3b46748fc1da662c1d4c48" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41227 } } peers { permanent_uuid: "6f1ec800ba244930917331cd412688da" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 44961 } } peers { permanent_uuid: "75ab19d1756044bd858010d7be60b1d1" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 33035 } }
I20260811 22:29:52.741145 29849 leader_election.cc:288] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 75cd5e101e3b46748fc1da662c1d4c48 (127.29.106.3:41227), 75ab19d1756044bd858010d7be60b1d1 (127.29.106.2:33035)
Aug 11 22:29:52 dist-test-slave-gjrz krb5kdc[26671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487391, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.1@KRBTEST.COM for kudu/127.29.106.2@KRBTEST.COM
Aug 11 22:29:52 dist-test-slave-gjrz krb5kdc[26671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487391, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.1@KRBTEST.COM for kudu/127.29.106.3@KRBTEST.COM
I20260811 22:29:52.765363 29862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:52.741613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:40823 (local address 127.29.106.3:41227)
0811 22:29:52.742114 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0811 22:29:52.742128 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0811 22:29:52.742177 (+ 49us) server_negotiation.cc:407] Connection header received
0811 22:29:52.744969 (+ 2792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:29:52.744981 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:29:52.745474 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:29:52.745835 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:29:52.748422 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:52.750291 (+ 1869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:52.752957 (+ 2666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:52.753941 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:52.754121 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:52.758706 (+ 4585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:29:52.758713 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:29:52.758725 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:29:52.758758 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:29:52.761738 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:29:52.762538 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:29:52.762546 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:29:52.762551 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:29:52.762714 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:29:52.763125 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:29:52.763128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:29:52.763130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:29:52.763627 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:29:52.763802 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0811 22:29:52.764622 (+ 820us) server_negotiation.cc:299] Negotiation successful
0811 22:29:52.764959 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":218,"threads_started":1}
I20260811 22:29:52.765360 29863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:52.741978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.2:33035 (local address 127.29.106.1:36461)
0811 22:29:52.742704 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:52.742724 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 22:29:52.742846 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:29:52.746010 (+ 3164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:29:52.746022 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:29:52.747130 (+ 1108us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:29:52.748216 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:29:52.748253 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:29:52.750496 (+ 2243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:29:52.750501 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:29:52.752829 (+ 2328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:29:52.752845 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:29:52.754196 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:29:52.754204 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:29:52.754544 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:52.755422 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:29:52.755453 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:29:52.758331 (+ 2878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:29:52.761865 (+ 3534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:29:52.761884 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:29:52.761899 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:29:52.762398 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:29:52.762809 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:29:52.762815 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:29:52.762819 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:29:52.763013 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:29:52.763764 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:29:52.763772 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:29:52.764056 (+ 284us) client_negotiation.cc:769] Sending connection context
0811 22:29:52.764373 (+ 317us) client_negotiation.cc:240] Negotiation successful
0811 22:29:52.764857 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"mutex_wait_us":354,"spinlock_wait_cycles":8064,"thread_start_us":386,"threads_started":1}
I20260811 22:29:52.765401 29868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:52.742395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:36461 (local address 127.29.106.2:33035)
0811 22:29:52.743014 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0811 22:29:52.743026 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 22:29:52.743069 (+ 43us) server_negotiation.cc:407] Connection header received
0811 22:29:52.745192 (+ 2123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:29:52.745201 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:29:52.745532 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:29:52.745835 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:29:52.748422 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:52.750291 (+ 1869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:52.752957 (+ 2666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:29:52.753941 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:29:52.754121 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:52.758610 (+ 4489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:29:52.758618 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:29:52.758630 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:29:52.758665 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:29:52.761730 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:29:52.762537 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:29:52.762544 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:29:52.762549 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:29:52.762703 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:29:52.763125 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:29:52.763129 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:29:52.763131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:29:52.763627 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:29:52.763805 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0811 22:29:52.764653 (+ 848us) server_negotiation.cc:299] Negotiation successful
0811 22:29:52.765037 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":249,"threads_started":1}
I20260811 22:29:52.765448 29858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:52.741328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.3:41227 (local address 127.29.106.1:40823)
0811 22:29:52.741826 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:52.741891 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0811 22:29:52.742489 (+ 598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:29:52.746010 (+ 3521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:29:52.746021 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:29:52.747130 (+ 1109us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:29:52.748216 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:29:52.748253 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:29:52.750493 (+ 2240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:29:52.750499 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:29:52.752829 (+ 2330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:29:52.752845 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:29:52.754159 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:29:52.754167 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:29:52.754539 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:29:52.755396 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:29:52.755436 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:29:52.758469 (+ 3033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:29:52.761869 (+ 3400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:29:52.761886 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:29:52.761899 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:29:52.762412 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:29:52.762823 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:29:52.762827 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:29:52.762829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:29:52.763012 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:29:52.763764 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:29:52.763771 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:29:52.764056 (+ 285us) client_negotiation.cc:769] Sending connection context
0811 22:29:52.764341 (+ 285us) client_negotiation.cc:240] Negotiation successful
0811 22:29:52.765029 (+ 688us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":515,"spinlock_wait_cycles":51712,"thread_start_us":251,"threads_started":1}
W20260811 22:29:52.768643 29057 leader_election.cc:341] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 75ab19d1756044bd858010d7be60b1d1 (127.29.106.2:33035): Not found: Tablet not found: e47e7811302a47d38df2164c4d056d7b
W20260811 22:29:52.768934 29057 leader_election.cc:341] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 75cd5e101e3b46748fc1da662c1d4c48 (127.29.106.3:41227): Not found: Tablet not found: e47e7811302a47d38df2164c4d056d7b
I20260811 22:29:52.769057 29057 leader_election.cc:302] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f1ec800ba244930917331cd412688da; no voters: 75ab19d1756044bd858010d7be60b1d1, 75cd5e101e3b46748fc1da662c1d4c48
I20260811 22:29:52.769420 29849 raft_consensus.cc:2726] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 22:29:53.037632 29870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:53.036758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.3:42629)
0811 22:29:53.037251 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:53.037393 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":240,"threads_started":1}
I20260811 22:29:53.154222 29858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:53.153743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.1:36933)
0811 22:29:53.153975 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:53.154076 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260811 22:29:53.592312 29875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:53.591440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.2:51295)
0811 22:29:53.592022 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:53.592145 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":261,"threads_started":1}
I20260811 22:29:54.039149 29881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:54.038438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.3:44483)
0811 22:29:54.038884 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:54.038991 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":228,"threads_started":1}
I20260811 22:29:54.155936 29884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:54.155109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.1:46223)
0811 22:29:54.155656 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:54.155749 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":252,"threads_started":1}
I20260811 22:29:54.593809 29885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:54.593135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.2:54523)
0811 22:29:54.593563 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:54.593644 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":225,"threads_started":1}
I20260811 22:29:54.763091 29895 raft_consensus.cc:482] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:29:54.763296 29895 raft_consensus.cc:504] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cd5e101e3b46748fc1da662c1d4c48" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41227 } } peers { permanent_uuid: "6f1ec800ba244930917331cd412688da" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 44961 } } peers { permanent_uuid: "75ab19d1756044bd858010d7be60b1d1" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 33035 } }
I20260811 22:29:54.763993 29895 leader_election.cc:288] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 75cd5e101e3b46748fc1da662c1d4c48 (127.29.106.3:41227), 75ab19d1756044bd858010d7be60b1d1 (127.29.106.2:33035)
W20260811 22:29:54.765054 29057 leader_election.cc:341] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 75cd5e101e3b46748fc1da662c1d4c48 (127.29.106.3:41227): Not found: Tablet not found: e47e7811302a47d38df2164c4d056d7b
W20260811 22:29:54.765256 29057 leader_election.cc:341] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 75ab19d1756044bd858010d7be60b1d1 (127.29.106.2:33035): Not found: Tablet not found: e47e7811302a47d38df2164c4d056d7b
I20260811 22:29:54.765403 29057 leader_election.cc:302] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f1ec800ba244930917331cd412688da; no voters: 75ab19d1756044bd858010d7be60b1d1, 75cd5e101e3b46748fc1da662c1d4c48
I20260811 22:29:54.765709 29895 raft_consensus.cc:2726] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 22:29:55.040644 29896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:55.039944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.3:41247)
0811 22:29:55.040392 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:55.040485 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":240,"threads_started":1}
I20260811 22:29:55.157646 29899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:55.156778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.1:40145)
0811 22:29:55.157310 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:55.157433 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":304,"threads_started":1}
I20260811 22:29:55.595235 29900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:55.594588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.2:49105)
0811 22:29:55.594990 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:55.595065 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":200,"threads_started":1}
I20260811 22:29:56.042130 29903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:56.041446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.3:34481)
0811 22:29:56.041872 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:56.041960 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":225,"threads_started":1}
I20260811 22:29:56.159431 29905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:56.158710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.1:45819)
0811 22:29:56.159133 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:56.159214 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":224,"threads_started":1}
I20260811 22:29:56.596678 29906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:56.596028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.2:37443)
0811 22:29:56.596434 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:56.596513 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":200,"threads_started":1}
I20260811 22:29:57.043609 29907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:57.042919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.3:41353)
0811 22:29:57.043363 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:57.043454 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":219,"threads_started":1}
I20260811 22:29:57.160990 29908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:57.160232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.1:48639)
0811 22:29:57.160729 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:57.160811 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":296,"threads_started":1}
I20260811 22:29:57.598129 29909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:57.597412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.2:55551)
0811 22:29:57.597863 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:57.597948 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":240,"threads_started":1}
I20260811 22:29:57.609381 29910 raft_consensus.cc:482] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:29:57.609978 29910 raft_consensus.cc:504] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cd5e101e3b46748fc1da662c1d4c48" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41227 } } peers { permanent_uuid: "6f1ec800ba244930917331cd412688da" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 44961 } } peers { permanent_uuid: "75ab19d1756044bd858010d7be60b1d1" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 33035 } }
I20260811 22:29:57.610615 29910 leader_election.cc:288] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 75cd5e101e3b46748fc1da662c1d4c48 (127.29.106.3:41227), 75ab19d1756044bd858010d7be60b1d1 (127.29.106.2:33035)
W20260811 22:29:57.611486 29057 leader_election.cc:341] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 75cd5e101e3b46748fc1da662c1d4c48 (127.29.106.3:41227): Not found: Tablet not found: e47e7811302a47d38df2164c4d056d7b
W20260811 22:29:57.611698 29057 leader_election.cc:341] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 75ab19d1756044bd858010d7be60b1d1 (127.29.106.2:33035): Not found: Tablet not found: e47e7811302a47d38df2164c4d056d7b
I20260811 22:29:57.611801 29057 leader_election.cc:302] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f1ec800ba244930917331cd412688da; no voters: 75ab19d1756044bd858010d7be60b1d1, 75cd5e101e3b46748fc1da662c1d4c48
I20260811 22:29:57.612066 29910 raft_consensus.cc:2726] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 22:29:58.045050 29911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:58.044381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.3:38289)
0811 22:29:58.044810 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:58.044887 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":236,"threads_started":1}
I20260811 22:29:58.162698 29915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:58.161860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.1:51365)
0811 22:29:58.162377 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:58.162486 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":245,"threads_started":1}
I20260811 22:29:58.599716 29916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:58.598889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.2:48943)
0811 22:29:58.599446 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:58.599538 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":236,"threads_started":1}
I20260811 22:29:59.046619 29917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:59.045811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.3:48399)
0811 22:29:59.046323 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:59.046412 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":219,"threads_started":1}
I20260811 22:29:59.164297 29918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:59.163591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.1:44309)
0811 22:29:59.164030 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:59.164137 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":216,"threads_started":1}
I20260811 22:29:59.601233 29919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:29:59.600525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.2:55797)
0811 22:29:59.600934 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0811 22:29:59.601061 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":203,"threads_started":1}
I20260811 22:29:59.954049 29920 raft_consensus.cc:482] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:29:59.954276 29920 raft_consensus.cc:504] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cd5e101e3b46748fc1da662c1d4c48" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41227 } } peers { permanent_uuid: "6f1ec800ba244930917331cd412688da" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 44961 } } peers { permanent_uuid: "75ab19d1756044bd858010d7be60b1d1" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 33035 } }
I20260811 22:29:59.954860 29920 leader_election.cc:288] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 75cd5e101e3b46748fc1da662c1d4c48 (127.29.106.3:41227), 75ab19d1756044bd858010d7be60b1d1 (127.29.106.2:33035)
W20260811 22:29:59.955763 29057 leader_election.cc:341] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 75cd5e101e3b46748fc1da662c1d4c48 (127.29.106.3:41227): Not found: Tablet not found: e47e7811302a47d38df2164c4d056d7b
W20260811 22:29:59.955987 29057 leader_election.cc:341] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 75ab19d1756044bd858010d7be60b1d1 (127.29.106.2:33035): Not found: Tablet not found: e47e7811302a47d38df2164c4d056d7b
I20260811 22:29:59.956094 29057 leader_election.cc:302] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f1ec800ba244930917331cd412688da; no voters: 75ab19d1756044bd858010d7be60b1d1, 75cd5e101e3b46748fc1da662c1d4c48
I20260811 22:29:59.956352 29920 raft_consensus.cc:2726] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 22:30:00.048106 29921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:00.047472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.3:59691)
0811 22:30:00.047862 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:00.047938 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":201,"threads_started":1}
I20260811 22:30:00.166038 29922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:00.165189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.1:59155)
0811 22:30:00.165704 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:00.165800 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":258,"threads_started":1}
I20260811 22:30:00.602730 29923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:00.602020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.2:41677)
0811 22:30:00.602490 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:00.602577 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":220,"threads_started":1}
I20260811 22:30:01.049631 29924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:01.048897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.3:56481)
0811 22:30:01.049390 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:01.049477 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":253,"threads_started":1}
I20260811 22:30:01.167685 29925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:01.166973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.1:48405)
0811 22:30:01.167430 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:01.167511 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":223,"threads_started":1}
I20260811 22:30:01.604238 29926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:01.603546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.2:47251)
0811 22:30:01.603965 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:01.604062 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":212,"threads_started":1}
I20260811 22:30:01.826941 29928 raft_consensus.cc:482] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:30:01.827153 29928 raft_consensus.cc:504] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cd5e101e3b46748fc1da662c1d4c48" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41227 } } peers { permanent_uuid: "6f1ec800ba244930917331cd412688da" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 44961 } } peers { permanent_uuid: "75ab19d1756044bd858010d7be60b1d1" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 33035 } }
I20260811 22:30:01.827821 29928 leader_election.cc:288] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 75cd5e101e3b46748fc1da662c1d4c48 (127.29.106.3:41227), 75ab19d1756044bd858010d7be60b1d1 (127.29.106.2:33035)
W20260811 22:30:01.828751 29057 leader_election.cc:341] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 75cd5e101e3b46748fc1da662c1d4c48 (127.29.106.3:41227): Not found: Tablet not found: e47e7811302a47d38df2164c4d056d7b
W20260811 22:30:01.828945 29057 leader_election.cc:341] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 75ab19d1756044bd858010d7be60b1d1 (127.29.106.2:33035): Not found: Tablet not found: e47e7811302a47d38df2164c4d056d7b
I20260811 22:30:01.829036 29057 leader_election.cc:302] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f1ec800ba244930917331cd412688da; no voters: 75ab19d1756044bd858010d7be60b1d1, 75cd5e101e3b46748fc1da662c1d4c48
I20260811 22:30:01.829260 29928 raft_consensus.cc:2726] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 22:30:02.051182 29929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:02.050494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.3:35743)
0811 22:30:02.050915 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:02.051003 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":229,"threads_started":1}
I20260811 22:30:02.169484 29931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:02.168651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.1:36685)
0811 22:30:02.169131 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:02.169223 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":235,"threads_started":1}
I20260811 22:30:02.605723 29932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:02.605037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.2:41915)
0811 22:30:02.605484 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:02.605565 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":214,"threads_started":1}
I20260811 22:30:03.052814 29935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:03.052019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.3:44455)
0811 22:30:03.052545 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:03.052633 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":275,"threads_started":1}
I20260811 22:30:03.171159 29936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:03.170348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.1:47007)
0811 22:30:03.170833 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:03.170964 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":249,"threads_started":1}
I20260811 22:30:03.607184 29937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:03.606517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.2:53273)
0811 22:30:03.606940 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:03.607018 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":223,"threads_started":1}
I20260811 22:30:04.054406 29938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:04.053682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.3:33019)
0811 22:30:04.054142 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:04.054224 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":244,"threads_started":1}
I20260811 22:30:04.172752 29942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:04.172006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.1:43159)
0811 22:30:04.172470 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:04.172574 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":238,"threads_started":1}
I20260811 22:30:04.608767 29947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:04.608028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.2:54003)
0811 22:30:04.608503 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:04.608605 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":231,"threads_started":1}
I20260811 22:30:05.056241 29949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:05.055380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.3:46641)
0811 22:30:05.055935 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:05.056058 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":270,"threads_started":1}
I20260811 22:30:05.077687 29950 raft_consensus.cc:482] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:30:05.077857 29950 raft_consensus.cc:504] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75cd5e101e3b46748fc1da662c1d4c48" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41227 } } peers { permanent_uuid: "6f1ec800ba244930917331cd412688da" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 44961 } } peers { permanent_uuid: "75ab19d1756044bd858010d7be60b1d1" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 33035 } }
I20260811 22:30:05.078503 29950 leader_election.cc:288] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 75cd5e101e3b46748fc1da662c1d4c48 (127.29.106.3:41227), 75ab19d1756044bd858010d7be60b1d1 (127.29.106.2:33035)
W20260811 22:30:05.079411 29057 leader_election.cc:341] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 75cd5e101e3b46748fc1da662c1d4c48 (127.29.106.3:41227): Not found: Tablet not found: e47e7811302a47d38df2164c4d056d7b
W20260811 22:30:05.079643 29057 leader_election.cc:341] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 75ab19d1756044bd858010d7be60b1d1 (127.29.106.2:33035): Not found: Tablet not found: e47e7811302a47d38df2164c4d056d7b
I20260811 22:30:05.079743 29057 leader_election.cc:302] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f1ec800ba244930917331cd412688da; no voters: 75ab19d1756044bd858010d7be60b1d1, 75cd5e101e3b46748fc1da662c1d4c48
I20260811 22:30:05.079991 29950 raft_consensus.cc:2726] T e47e7811302a47d38df2164c4d056d7b P 6f1ec800ba244930917331cd412688da [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 22:30:05.174623 29951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:05.173729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.1:55305)
0811 22:30:05.174272 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:05.174401 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":261,"threads_started":1}
I20260811 22:30:05.610488 29954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:05.609689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.2:54653)
0811 22:30:05.610191 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:05.610285 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":256,"threads_started":1}
I20260811 22:30:06.057971 29955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:06.057136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.3:51461)
0811 22:30:06.057660 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:06.057759 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":274,"threads_started":1}
I20260811 22:30:06.176717 29956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:06.175989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.1:37087)
0811 22:30:06.176440 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:06.176540 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":224,"threads_started":1}
I20260811 22:30:06.612386 29957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:06.611609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.2:36161)
0811 22:30:06.612105 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:06.612215 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":224,"threads_started":1}
I20260811 22:30:07.046306 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 28919
I20260811 22:30:07.059731 29959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:07.059006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:45241 (local address 127.29.106.3:35333)
0811 22:30:07.059477 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:07.059574 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.62:45241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":264,"threads_started":1}
I20260811 22:30:07.061349 29958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:07.047356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35568 (local address 127.29.106.1:44961)
0811 22:30:07.047991 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:07.048001 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:07.048030 (+ 29us) server_negotiation.cc:407] Connection header received
0811 22:30:07.048105 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:07.048112 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:07.048323 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:07.048452 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:07.050095 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:07.051110 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:07.052472 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:07.053035 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:07.053144 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:07.055667 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:07.055679 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:07.055694 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:30:07.055726 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:07.058427 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:07.059167 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:07.059176 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:07.059178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:07.059349 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:07.059751 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:07.059754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:07.059757 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:07.060200 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:07.060491 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:07.060805 (+ 314us) server_negotiation.cc:299] Negotiation successful
0811 22:30:07.061075 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":335,"threads_started":1}
I20260811 22:30:07.062562 29282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:30:07.150178 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 28919
I20260811 22:30:07.174022 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 29308
I20260811 22:30:07.187134 29965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:07.174773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55808 (local address 127.29.106.2:33035)
0811 22:30:07.175371 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:07.175375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:07.175393 (+ 18us) server_negotiation.cc:407] Connection header received
0811 22:30:07.175457 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:07.175461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:07.175521 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:07.175610 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:07.177094 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:07.177893 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:07.179116 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:07.179667 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:07.179768 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:07.182367 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:07.182371 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:07.182374 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:30:07.182408 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:07.184846 (+ 2438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:07.185461 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:07.185464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:07.185466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:07.185607 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:07.185984 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:07.185987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:07.185989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:07.186255 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:07.186440 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:07.186656 (+ 216us) server_negotiation.cc:299] Negotiation successful
0811 22:30:07.186819 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":329,"threads_started":1}
I20260811 22:30:07.187851 29564 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:30:07.268745 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 29308
I20260811 22:30:07.295610 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 29576
I20260811 22:30:07.309293 29969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:07.296387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60376 (local address 127.29.106.3:41227)
0811 22:30:07.296963 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:07.296968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:07.296984 (+ 16us) server_negotiation.cc:407] Connection header received
0811 22:30:07.297039 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:07.297043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:07.297110 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:07.297207 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:07.298635 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:07.299500 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:07.300754 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:07.301206 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:07.301319 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:07.304035 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:07.304040 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:07.304044 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:30:07.304074 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:07.306653 (+ 2579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:07.307337 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:07.307343 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:07.307347 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:07.307490 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:07.307870 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:07.307875 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:07.307878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:07.308152 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:07.308407 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:07.308692 (+ 285us) server_negotiation.cc:299] Negotiation successful
0811 22:30:07.308924 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":302,"threads_started":1}
I20260811 22:30:07.309957 29827 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:30:07.392606 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 29576
2026-08-11T22:30:07Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64686 ms)
[----------] 30 tests from SecurityITest (301504 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:30:07 dist-test-slave-gjrz krb5kdc[29984](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:30:07 dist-test-slave-gjrz krb5kdc[29984](info): set up 2 sockets
Aug 11 22:30:07 dist-test-slave-gjrz krb5kdc[29984](info): commencing operation
krb5kdc: starting...
W20260811 22:30:08.657212 30120 mini_kdc.cc:121] Time spent starting KDC: real 1.216s user 0.000s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Aug 11 22:30:08 dist-test-slave-gjrz krb5kdc[29984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:30:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:30:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.106.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
I20260811 22:30:08.916394 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:37499
--webserver_interface=127.29.106.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:35625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:37499
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:30:09.243626 30011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:30:09.243887 30011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:30:09.244022 30011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:30:09.251031 30011 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:30:09.251127 30011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:30:09.251183 30011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:30:09.251281 30011 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:30:09.251345 30011 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:30:09.260802 30011 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:35625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:37499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:37499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.106.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:30:09.262059 30011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:30:09.263365 30011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:30:09.274493 30017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:09.274561 30021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:09.274529 30018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:30:09.275318 30011 server_base.cc:1034] running on GCE node
I20260811 22:30:09.276064 30011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:30:09.277431 30011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:30:09.278647 30011 hybrid_clock.cc:648] HybridClock initialized: now 1786487409278588 us; error 46 us; skew 500 ppm
I20260811 22:30:09.278982 30011 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:30:09 dist-test-slave-gjrz krb5kdc[29984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487409, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:30:09.282022 30011 init.cc:377] Logged in from keytab as kudu/127.29.106.62@KRBTEST.COM (short username kudu)
I20260811 22:30:09.283422 30011 webserver.cc:460] Webserver started at http://127.29.106.62:46643/ using document root <none> and password file <none>
I20260811 22:30:09.284062 30011 fs_manager.cc:359] Metadata directory not provided
I20260811 22:30:09.284181 30011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:30:09.284461 30011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:30:09.287377 30011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "3a93f855414a41f589c8f94b6fa4f266"
format_stamp: "Formatted at 2026-08-11 22:30:09 on dist-test-slave-gjrz"
server_key: "7d04a0d9afa220f38da06717ae2f2a1b"
server_key_iv: "1abd99bb4d3100e2a73fd6833e13a0b5"
server_key_version: "encryptionkey@0"
I20260811 22:30:09.288120 30011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "3a93f855414a41f589c8f94b6fa4f266"
format_stamp: "Formatted at 2026-08-11 22:30:09 on dist-test-slave-gjrz"
server_key: "7d04a0d9afa220f38da06717ae2f2a1b"
server_key_iv: "1abd99bb4d3100e2a73fd6833e13a0b5"
server_key_version: "encryptionkey@0"
I20260811 22:30:09.292989 30011 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260811 22:30:09.296337 30037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:09.297518 30011 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260811 22:30:09.297673 30011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "3a93f855414a41f589c8f94b6fa4f266"
format_stamp: "Formatted at 2026-08-11 22:30:09 on dist-test-slave-gjrz"
server_key: "7d04a0d9afa220f38da06717ae2f2a1b"
server_key_iv: "1abd99bb4d3100e2a73fd6833e13a0b5"
server_key_version: "encryptionkey@0"
I20260811 22:30:09.297843 30011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:30:09.327562 30011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:30:09.331763 30011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:30:09.332022 30011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:30:09.349821 30011 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.62:37499
I20260811 22:30:09.349844 30159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.62:37499 every 8 connection(s)
I20260811 22:30:09.351132 30011 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:30:09.354874 30160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:30:09.359184 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 30011
I20260811 22:30:09.359412 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:30:09.360062 30120 external_mini_cluster.cc:1428] Setting key 572e8af385880ad9a78a4d3d84050031
I20260811 22:30:09.365417 30160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a93f855414a41f589c8f94b6fa4f266: Bootstrap starting.
I20260811 22:30:09.369001 30160 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a93f855414a41f589c8f94b6fa4f266: Neither blocks nor log segments found. Creating new log.
I20260811 22:30:09.370234 30160 log.cc:824] T 00000000000000000000000000000000 P 3a93f855414a41f589c8f94b6fa4f266: Log is configured to *not* fsync() on all Append() calls
I20260811 22:30:09.373463 30160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a93f855414a41f589c8f94b6fa4f266: No bootstrap required, opened a new log
Aug 11 22:30:09 dist-test-slave-gjrz krb5kdc[29984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:30:09.381108 30160 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a93f855414a41f589c8f94b6fa4f266 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a93f855414a41f589c8f94b6fa4f266" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 37499 } }
I20260811 22:30:09.381457 30160 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a93f855414a41f589c8f94b6fa4f266 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:30:09.381560 30160 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a93f855414a41f589c8f94b6fa4f266 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a93f855414a41f589c8f94b6fa4f266, State: Initialized, Role: FOLLOWER
I20260811 22:30:09.382159 30160 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a93f855414a41f589c8f94b6fa4f266 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a93f855414a41f589c8f94b6fa4f266" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 37499 } }
I20260811 22:30:09.382404 30160 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a93f855414a41f589c8f94b6fa4f266 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:30:09.382565 30160 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a93f855414a41f589c8f94b6fa4f266 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:30:09.382750 30160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a93f855414a41f589c8f94b6fa4f266 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:30:09.382930 30167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:09.361929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59020 (local address 127.29.106.62:37499)
0811 22:30:09.362800 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:09.362811 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:09.362853 (+ 42us) server_negotiation.cc:407] Connection header received
0811 22:30:09.364580 (+ 1727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:09.364601 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:09.364961 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:09.365259 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:09.367083 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:09.368410 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:09.370261 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:09.371163 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:09.371387 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:09.375915 (+ 4528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:09.375922 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:09.375936 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:30:09.375975 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:09.378987 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:09.379676 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:09.379683 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:09.379689 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:09.379860 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:09.380227 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:09.380234 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:09.380236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:09.380709 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:09.380971 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:09.381411 (+ 440us) server_negotiation.cc:299] Negotiation successful
0811 22:30:09.381806 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":371,"threads_started":1}
I20260811 22:30:09.385054 30160 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a93f855414a41f589c8f94b6fa4f266 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a93f855414a41f589c8f94b6fa4f266" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 37499 } }
I20260811 22:30:09.385519 30160 leader_election.cc:302] T 00000000000000000000000000000000 P 3a93f855414a41f589c8f94b6fa4f266 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a93f855414a41f589c8f94b6fa4f266; no voters:
I20260811 22:30:09.386065 30160 leader_election.cc:288] T 00000000000000000000000000000000 P 3a93f855414a41f589c8f94b6fa4f266 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:30:09.386175 30169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a93f855414a41f589c8f94b6fa4f266 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:30:09.386799 30169 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a93f855414a41f589c8f94b6fa4f266 [term 1 LEADER]: Becoming Leader. State: Replica: 3a93f855414a41f589c8f94b6fa4f266, State: Running, Role: LEADER
I20260811 22:30:09.387308 30169 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a93f855414a41f589c8f94b6fa4f266 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a93f855414a41f589c8f94b6fa4f266" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 37499 } }
I20260811 22:30:09.388023 30160 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a93f855414a41f589c8f94b6fa4f266 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:30:09.389818 30175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a93f855414a41f589c8f94b6fa4f266 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a93f855414a41f589c8f94b6fa4f266. Latest consensus state: current_term: 1 leader_uuid: "3a93f855414a41f589c8f94b6fa4f266" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a93f855414a41f589c8f94b6fa4f266" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 37499 } } }
I20260811 22:30:09.390136 30175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a93f855414a41f589c8f94b6fa4f266 [sys.catalog]: This master's current role is: LEADER
I20260811 22:30:09.390980 30180 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:30:09.390821 30173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a93f855414a41f589c8f94b6fa4f266 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a93f855414a41f589c8f94b6fa4f266" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a93f855414a41f589c8f94b6fa4f266" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 37499 } } }
I20260811 22:30:09.392814 30173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a93f855414a41f589c8f94b6fa4f266 [sys.catalog]: This master's current role is: LEADER
I20260811 22:30:09.396296 30180 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:30:09.405687 30180 catalog_manager.cc:1347] Generated new cluster ID: f154b6efefd14169b65eb61cfa57c3af
I20260811 22:30:09.405783 30180 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:30:09.437350 30180 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:30:09.438589 30180 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:30:09.445113 30180 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a93f855414a41f589c8f94b6fa4f266: Generated new TSK 0
I20260811 22:30:09.445809 30180 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.106.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab.
Entry for principal kudu/127.29.106.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab.
I20260811 22:30:09.518586 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.1:0
--local_ip_for_outbound_sockets=127.29.106.1
--webserver_interface=127.29.106.1
--webserver_port=0
--tserver_master_addrs=127.29.106.62:37499
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab
--principal=kudu/127.29.106.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:35625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:30:09.849226 30214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:30:09.849498 30214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:30:09.849627 30214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:30:09.856482 30214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:30:09.856590 30214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:30:09.856736 30214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.1
I20260811 22:30:09.865792 30214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:35625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab
--principal=kudu/127.29.106.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.106.1
--webserver_port=0
--tserver_master_addrs=127.29.106.62:37499
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.1
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:30:09.866930 30214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:30:09.868183 30214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:30:09.880645 30228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:09.880586 30223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:09.880590 30222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:30:09.881572 30214 server_base.cc:1034] running on GCE node
I20260811 22:30:09.882166 30214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:30:09.883029 30214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:30:09.884225 30214 hybrid_clock.cc:648] HybridClock initialized: now 1786487409884164 us; error 54 us; skew 500 ppm
I20260811 22:30:09.884516 30214 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:30:09 dist-test-slave-gjrz krb5kdc[29984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487409, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:30:09.887465 30214 init.cc:377] Logged in from keytab as kudu/127.29.106.1@KRBTEST.COM (short username kudu)
I20260811 22:30:09.888809 30214 webserver.cc:460] Webserver started at http://127.29.106.1:40799/ using document root <none> and password file <none>
I20260811 22:30:09.889500 30214 fs_manager.cc:359] Metadata directory not provided
I20260811 22:30:09.889617 30214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:30:09.889904 30214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:30:09.892652 30214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-0/data/instance:
uuid: "dcd5e96211a04b37aa250ea0115bf45d"
format_stamp: "Formatted at 2026-08-11 22:30:09 on dist-test-slave-gjrz"
server_key: "ff6c1235783a86a147c0bfd693a90a7f"
server_key_iv: "ea9b158fa370c19ccff33ef00a269bb4"
server_key_version: "encryptionkey@0"
I20260811 22:30:09.893394 30214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance:
uuid: "dcd5e96211a04b37aa250ea0115bf45d"
format_stamp: "Formatted at 2026-08-11 22:30:09 on dist-test-slave-gjrz"
server_key: "ff6c1235783a86a147c0bfd693a90a7f"
server_key_iv: "ea9b158fa370c19ccff33ef00a269bb4"
server_key_version: "encryptionkey@0"
I20260811 22:30:09.898252 30214 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260811 22:30:09.901525 30242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:09.902802 30214 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260811 22:30:09.902945 30214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-0/wal
uuid: "dcd5e96211a04b37aa250ea0115bf45d"
format_stamp: "Formatted at 2026-08-11 22:30:09 on dist-test-slave-gjrz"
server_key: "ff6c1235783a86a147c0bfd693a90a7f"
server_key_iv: "ea9b158fa370c19ccff33ef00a269bb4"
server_key_version: "encryptionkey@0"
I20260811 22:30:09.903124 30214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:30:09.917866 30214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:30:09.921849 30214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:30:09.922119 30214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:30:09.922988 30214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:30:09.924285 30214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:30:09.924363 30214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:09.924484 30214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:30:09.924547 30214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:09.954757 30214 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.1:34143
I20260811 22:30:09.954802 30450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.1:34143 every 8 connection(s)
I20260811 22:30:09.955938 30214 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
I20260811 22:30:09.966302 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 30214
I20260811 22:30:09.966500 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance
I20260811 22:30:09.967129 30120 external_mini_cluster.cc:1428] Setting key d546381f5210ac8b6dea95fcb9832055
Aug 11 22:30:09 dist-test-slave-gjrz krb5kdc[29984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487409, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.1@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:30:09.974642 30453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:09.958194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:38959 (local address 127.29.106.62:37499)
0811 22:30:09.958637 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:09.958644 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:09.959532 (+ 888us) server_negotiation.cc:407] Connection header received
0811 22:30:09.961270 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:09.961274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:09.961341 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:09.961433 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:09.962819 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:09.963613 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:09.965035 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:09.965413 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:09.965503 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:09.968825 (+ 3322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:09.968830 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:09.968834 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:30:09.968864 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:09.971844 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:09.972522 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:09.972527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:09.972530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:09.972671 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:09.973042 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:09.973046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:09.973050 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:09.973367 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:09.973499 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:09.974105 (+ 606us) server_negotiation.cc:299] Negotiation successful
0811 22:30:09.974315 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":200,"threads_started":1}
I20260811 22:30:09.975378 30454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:09.958657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:37499 (local address 127.29.106.1:38959)
0811 22:30:09.959379 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:09.959425 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:09.961043 (+ 1618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:09.961639 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:09.961647 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:09.962126 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:30:09.962672 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:09.962690 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:09.963737 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:09.963741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:09.964927 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:09.964939 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:09.965590 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:09.965595 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:09.965788 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:09.966410 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:30:09.966443 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:30:09.968625 (+ 2182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:30:09.971990 (+ 3365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:30:09.971996 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:30:09.972006 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:30:09.972404 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:30:09.972760 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:30:09.972763 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:30:09.972765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:30:09.972941 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:30:09.973479 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:30:09.973484 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:30:09.973756 (+ 272us) client_negotiation.cc:769] Sending connection context
0811 22:30:09.973971 (+ 215us) client_negotiation.cc:240] Negotiation successful
0811 22:30:09.974306 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":276,"threads_started":1}
I20260811 22:30:09.976759 30451 heartbeater.cc:347] Connected to a master server at 127.29.106.62:37499
I20260811 22:30:09.977054 30451 heartbeater.cc:464] Registering TS with master...
I20260811 22:30:09.977835 30451 heartbeater.cc:511] Master 127.29.106.62:37499 requested a full tablet report, sending...
I20260811 22:30:09.979887 30075 ts_manager.cc:194] Registered new tserver with Master: dcd5e96211a04b37aa250ea0115bf45d (127.29.106.1:34143)
I20260811 22:30:09.981246 30075 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.1@KRBTEST.COM'} at 127.29.106.1:38959
WARNING: no policy specified for kudu/127.29.106.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab.
Entry for principal kudu/127.29.106.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab.
I20260811 22:30:10.027948 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.2:0
--local_ip_for_outbound_sockets=127.29.106.2
--webserver_interface=127.29.106.2
--webserver_port=0
--tserver_master_addrs=127.29.106.62:37499
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab
--principal=kudu/127.29.106.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:35625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:30:10.350389 30459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:30:10.350715 30459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:30:10.350869 30459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:30:10.361871 30459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:30:10.362011 30459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:30:10.362183 30459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.2
I20260811 22:30:10.374780 30459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:35625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab
--principal=kudu/127.29.106.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.106.2
--webserver_port=0
--tserver_master_addrs=127.29.106.62:37499
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.2
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:30:10.375872 30459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:30:10.377023 30459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:30:10.388890 30467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:10.388926 30470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:10.388949 30468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:30:10.389947 30459 server_base.cc:1034] running on GCE node
I20260811 22:30:10.390476 30459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:30:10.391258 30459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:30:10.392481 30459 hybrid_clock.cc:648] HybridClock initialized: now 1786487410392430 us; error 47 us; skew 500 ppm
I20260811 22:30:10.392758 30459 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:30:10 dist-test-slave-gjrz krb5kdc[29984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487410, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:30:10.395787 30459 init.cc:377] Logged in from keytab as kudu/127.29.106.2@KRBTEST.COM (short username kudu)
I20260811 22:30:10.397125 30459 webserver.cc:460] Webserver started at http://127.29.106.2:39829/ using document root <none> and password file <none>
I20260811 22:30:10.397724 30459 fs_manager.cc:359] Metadata directory not provided
I20260811 22:30:10.397840 30459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:30:10.398137 30459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:30:10.400863 30459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-1/data/instance:
uuid: "8259d78358174d538790ca8a6ad07798"
format_stamp: "Formatted at 2026-08-11 22:30:10 on dist-test-slave-gjrz"
server_key: "3e41c976a9fa28118d4b68cb1e4d9288"
server_key_iv: "8bfec8e4140b420dd15f116e5c2412fa"
server_key_version: "encryptionkey@0"
I20260811 22:30:10.401634 30459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance:
uuid: "8259d78358174d538790ca8a6ad07798"
format_stamp: "Formatted at 2026-08-11 22:30:10 on dist-test-slave-gjrz"
server_key: "3e41c976a9fa28118d4b68cb1e4d9288"
server_key_iv: "8bfec8e4140b420dd15f116e5c2412fa"
server_key_version: "encryptionkey@0"
I20260811 22:30:10.406474 30459 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 22:30:10.409776 30480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:10.410941 30459 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 22:30:10.411083 30459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-1/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-1/wal
uuid: "8259d78358174d538790ca8a6ad07798"
format_stamp: "Formatted at 2026-08-11 22:30:10 on dist-test-slave-gjrz"
server_key: "3e41c976a9fa28118d4b68cb1e4d9288"
server_key_iv: "8bfec8e4140b420dd15f116e5c2412fa"
server_key_version: "encryptionkey@0"
I20260811 22:30:10.411248 30459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:30:10.423776 30459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:30:10.427640 30459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:30:10.427891 30459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:30:10.428714 30459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:30:10.429900 30459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:30:10.429983 30459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:10.430097 30459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:30:10.430161 30459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:10.459303 30459 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.2:37401
I20260811 22:30:10.459347 30696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.2:37401 every 8 connection(s)
I20260811 22:30:10.460460 30459 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
I20260811 22:30:10.462710 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 30459
I20260811 22:30:10.462889 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance
I20260811 22:30:10.463442 30120 external_mini_cluster.cc:1428] Setting key 146be35c83d0023ba76142e13467b8a2
Aug 11 22:30:10 dist-test-slave-gjrz krb5kdc[29984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487410, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.2@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:30:10.480494 30453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:10.462847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:42269 (local address 127.29.106.62:37499)
0811 22:30:10.463053 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:10.463060 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:10.464213 (+ 1153us) server_negotiation.cc:407] Connection header received
0811 22:30:10.466022 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:10.466027 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:10.466106 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:10.466206 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:10.467765 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:10.468695 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:10.470290 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:10.470856 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:10.470977 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:10.474509 (+ 3532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:10.474515 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:10.474519 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:30:10.474564 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:10.477636 (+ 3072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:10.478292 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:10.478297 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:10.478301 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:10.478460 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:10.478903 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:10.478906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:10.478908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:10.479164 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:10.479316 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:10.480082 (+ 766us) server_negotiation.cc:299] Negotiation successful
0811 22:30:10.480251 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260811 22:30:10.481704 30703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:10.463235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:37499 (local address 127.29.106.2:42269)
0811 22:30:10.464045 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:10.464099 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:10.465764 (+ 1665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:10.466465 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:10.466473 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:10.466976 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:30:10.467593 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:10.467613 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:10.468849 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:10.468854 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:10.470147 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:10.470160 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:10.471116 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:10.471125 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:10.471448 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:10.472065 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:30:10.472101 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:30:10.474287 (+ 2186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:30:10.477765 (+ 3478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:30:10.477772 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:30:10.477783 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:30:10.478168 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:30:10.478562 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:30:10.478566 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:30:10.478570 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:30:10.478809 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:30:10.479294 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:30:10.479302 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:30:10.479710 (+ 408us) client_negotiation.cc:769] Sending connection context
0811 22:30:10.479986 (+ 276us) client_negotiation.cc:240] Negotiation successful
0811 22:30:10.480467 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":308,"threads_started":1}
I20260811 22:30:10.483080 30697 heartbeater.cc:347] Connected to a master server at 127.29.106.62:37499
I20260811 22:30:10.483392 30697 heartbeater.cc:464] Registering TS with master...
I20260811 22:30:10.484061 30697 heartbeater.cc:511] Master 127.29.106.62:37499 requested a full tablet report, sending...
I20260811 22:30:10.485394 30075 ts_manager.cc:194] Registered new tserver with Master: 8259d78358174d538790ca8a6ad07798 (127.29.106.2:37401)
I20260811 22:30:10.486171 30075 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.2@KRBTEST.COM'} at 127.29.106.2:42269
WARNING: no policy specified for kudu/127.29.106.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab.
Entry for principal kudu/127.29.106.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab.
I20260811 22:30:10.535876 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.3:0
--local_ip_for_outbound_sockets=127.29.106.3
--webserver_interface=127.29.106.3
--webserver_port=0
--tserver_master_addrs=127.29.106.62:37499
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab
--principal=kudu/127.29.106.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:35625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:30:10.856422 30714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:30:10.856685 30714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:30:10.856830 30714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:30:10.863762 30714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:30:10.863873 30714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:30:10.864033 30714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.3
I20260811 22:30:10.873373 30714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:35625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab
--principal=kudu/127.29.106.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.106.3
--webserver_port=0
--tserver_master_addrs=127.29.106.62:37499
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.3
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:30:10.874540 30714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:30:10.875829 30714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:30:10.888464 30737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:10.888590 30728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:10.888422 30730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:30:10.889752 30714 server_base.cc:1034] running on GCE node
I20260811 22:30:10.890333 30714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:30:10.891207 30714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:30:10.892478 30714 hybrid_clock.cc:648] HybridClock initialized: now 1786487410892430 us; error 35 us; skew 500 ppm
I20260811 22:30:10.892796 30714 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:30:10 dist-test-slave-gjrz krb5kdc[29984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487410, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:30:10.895903 30714 init.cc:377] Logged in from keytab as kudu/127.29.106.3@KRBTEST.COM (short username kudu)
I20260811 22:30:10.897332 30714 webserver.cc:460] Webserver started at http://127.29.106.3:41807/ using document root <none> and password file <none>
I20260811 22:30:10.897967 30714 fs_manager.cc:359] Metadata directory not provided
I20260811 22:30:10.898097 30714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:30:10.898396 30714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:30:10.901377 30714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-2/data/instance:
uuid: "bbbefaf172cb424da1e736876b03cf89"
format_stamp: "Formatted at 2026-08-11 22:30:10 on dist-test-slave-gjrz"
server_key: "08ebca85e98891a92fe10bf666de1eb4"
server_key_iv: "7eb7988bc385fb01a7f13c453fa039c4"
server_key_version: "encryptionkey@0"
I20260811 22:30:10.902150 30714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance:
uuid: "bbbefaf172cb424da1e736876b03cf89"
format_stamp: "Formatted at 2026-08-11 22:30:10 on dist-test-slave-gjrz"
server_key: "08ebca85e98891a92fe10bf666de1eb4"
server_key_iv: "7eb7988bc385fb01a7f13c453fa039c4"
server_key_version: "encryptionkey@0"
I20260811 22:30:10.907045 30714 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260811 22:30:10.910431 30750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:10.911677 30714 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:30:10.911835 30714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-2/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-2/wal
uuid: "bbbefaf172cb424da1e736876b03cf89"
format_stamp: "Formatted at 2026-08-11 22:30:10 on dist-test-slave-gjrz"
server_key: "08ebca85e98891a92fe10bf666de1eb4"
server_key_iv: "7eb7988bc385fb01a7f13c453fa039c4"
server_key_version: "encryptionkey@0"
I20260811 22:30:10.912019 30714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:30:10.925670 30714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:30:10.929621 30714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:30:10.929885 30714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:30:10.930730 30714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:30:10.932049 30714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:30:10.932127 30714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:10.932245 30714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:30:10.932303 30714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:10.962601 30714 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.3:41169
I20260811 22:30:10.962658 30951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.3:41169 every 8 connection(s)
I20260811 22:30:10.963805 30714 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
I20260811 22:30:10.969990 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 30714
I20260811 22:30:10.970151 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance
I20260811 22:30:10.970610 30120 external_mini_cluster.cc:1428] Setting key 22c1e0afc3a2bb8305cb21dc4cf4349e
Aug 11 22:30:10 dist-test-slave-gjrz krb5kdc[29984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487410, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.3@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:30:10.981835 30453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:10.966032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:38955 (local address 127.29.106.62:37499)
0811 22:30:10.966191 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:10.966195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:10.967223 (+ 1028us) server_negotiation.cc:407] Connection header received
0811 22:30:10.968964 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:10.968969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:10.969068 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:10.969156 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:10.970610 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:10.971537 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:10.973181 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:10.973591 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:10.973700 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:10.976931 (+ 3231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:10.976936 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:10.976940 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:30:10.976971 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:10.979237 (+ 2266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:10.979944 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:10.979948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:10.979951 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:10.980101 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:10.980480 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:10.980483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:10.980484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:10.980736 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:10.980843 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:10.981438 (+ 595us) server_negotiation.cc:299] Negotiation successful
0811 22:30:10.981598 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260811 22:30:10.982857 30960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:10.966371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:37499 (local address 127.29.106.3:38955)
0811 22:30:10.967054 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:10.967091 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:10.968712 (+ 1621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:10.969356 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:10.969363 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:10.969859 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:30:10.970431 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:10.970458 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:10.971683 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:10.971686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:10.973044 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:10.973056 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:10.973799 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:10.973807 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:10.974034 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:10.974621 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:30:10.974642 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:30:10.976746 (+ 2104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:30:10.979409 (+ 2663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:30:10.979416 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:30:10.979427 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:30:10.979832 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:30:10.980203 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:30:10.980206 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:30:10.980209 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:30:10.980399 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:30:10.980840 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:30:10.980846 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:30:10.981151 (+ 305us) client_negotiation.cc:769] Sending connection context
0811 22:30:10.981361 (+ 210us) client_negotiation.cc:240] Negotiation successful
0811 22:30:10.981714 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":269,"threads_started":1}
I20260811 22:30:10.984194 30451 heartbeater.cc:503] Master 127.29.106.62:37499 was elected leader, sending a full tablet report...
I20260811 22:30:10.984552 30952 heartbeater.cc:347] Connected to a master server at 127.29.106.62:37499
I20260811 22:30:10.984845 30952 heartbeater.cc:464] Registering TS with master...
I20260811 22:30:10.985711 30952 heartbeater.cc:511] Master 127.29.106.62:37499 requested a full tablet report, sending...
I20260811 22:30:10.986976 30075 ts_manager.cc:194] Registered new tserver with Master: bbbefaf172cb424da1e736876b03cf89 (127.29.106.3:41169)
I20260811 22:30:10.987777 30075 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.3@KRBTEST.COM'} at 127.29.106.3:38955
I20260811 22:30:10.997535 30120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 22:30:11.013648 30453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:11.001853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36314 (local address 127.29.106.62:37499)
0811 22:30:11.002095 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:11.002102 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:11.002403 (+ 301us) server_negotiation.cc:407] Connection header received
0811 22:30:11.002531 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:11.002535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:11.002593 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:11.002664 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:11.004008 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:11.004692 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:11.006003 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:11.006422 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:11.006509 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:11.008958 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:11.008963 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:11.008966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:30:11.008994 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:11.011468 (+ 2474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:11.012064 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:11.012067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:11.012069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:11.012188 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:11.012462 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:11.012465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:11.012466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:11.012757 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:11.012932 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:11.013238 (+ 306us) server_negotiation.cc:299] Negotiation successful
0811 22:30:11.013391 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260811 22:30:11.017346 30075 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36314:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:30:11.023932 30075 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:30:11.059109 30986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:11.035166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.3:41169 (local address 127.0.0.1:45448)
0811 22:30:11.051714 (+ 16548us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:11.051731 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:11.051859 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:11.052734 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:11.052742 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:11.052767 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:30:11.053148 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:11.053161 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:11.054818 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:11.054823 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:11.056788 (+ 1965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:11.056805 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:11.058123 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:11.058130 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:11.058300 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:11.058355 (+ 55us) client_negotiation.cc:769] Sending connection context
0811 22:30:11.058482 (+ 127us) client_negotiation.cc:240] Negotiation successful
0811 22:30:11.058666 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16431,"mutex_wait_us":26,"thread_start_us":9992,"threads_started":1}
I20260811 22:30:11.059151 30985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:11.035471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45448 (local address 127.29.106.3:41169)
0811 22:30:11.051892 (+ 16421us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:11.051900 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:11.051946 (+ 46us) server_negotiation.cc:407] Connection header received
0811 22:30:11.052046 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:11.052053 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:11.052260 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:11.052467 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:30:11.053357 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:11.054681 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:11.056987 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:11.057923 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:11.058046 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:11.058270 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:11.058701 (+ 431us) server_negotiation.cc:299] Negotiation successful
0811 22:30:11.058856 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16245,"thread_start_us":16066,"threads_started":1}
I20260811 22:30:11.059743 30981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:11.034948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.2:37401 (local address 127.0.0.1:56552)
0811 22:30:11.045564 (+ 10616us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:11.045622 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:11.045807 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:11.052157 (+ 6350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:11.052164 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:11.052194 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:30:11.052696 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:11.052712 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:11.054463 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:11.054468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:11.056601 (+ 2133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:11.056616 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:11.058498 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:11.058505 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:11.058679 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:11.058709 (+ 30us) client_negotiation.cc:769] Sending connection context
0811 22:30:11.058803 (+ 94us) client_negotiation.cc:240] Negotiation successful
0811 22:30:11.058936 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10344,"thread_start_us":16934,"threads_started":1}
I20260811 22:30:11.060592 30983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:11.035149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56552 (local address 127.29.106.2:37401)
0811 22:30:11.050864 (+ 15715us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:11.050873 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:11.050916 (+ 43us) server_negotiation.cc:407] Connection header received
0811 22:30:11.051016 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:11.051023 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:11.051302 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:11.052928 (+ 1626us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:30:11.053150 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:11.054327 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:11.056819 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:11.058263 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:11.058397 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:11.058584 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:11.060097 (+ 1513us) server_negotiation.cc:299] Negotiation successful
0811 22:30:11.060282 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15573,"thread_start_us":15466,"threads_started":1}
I20260811 22:30:11.060623 30979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:11.036620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:34143 (local address 127.0.0.1:36118)
0811 22:30:11.052485 (+ 15865us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:11.052504 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:11.052704 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:11.053506 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:11.053510 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:11.053535 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:30:11.053890 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:11.053902 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:11.055621 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:11.055626 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:11.057808 (+ 2182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:11.057823 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:11.059875 (+ 2052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:11.059881 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:11.060048 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:11.060078 (+ 30us) client_negotiation.cc:769] Sending connection context
0811 22:30:11.060159 (+ 81us) client_negotiation.cc:240] Negotiation successful
0811 22:30:11.060272 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15712,"mutex_wait_us":2,"thread_start_us":15343,"threads_started":1}
I20260811 22:30:11.061515 30987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:11.036961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36118 (local address 127.29.106.1:34143)
0811 22:30:11.051343 (+ 14382us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:11.051351 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:11.052600 (+ 1249us) server_negotiation.cc:407] Connection header received
0811 22:30:11.052895 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:11.052902 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:11.053088 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:11.053204 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:30:11.054061 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:11.055462 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:11.057979 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:11.059388 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:11.059517 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:11.059732 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:11.060823 (+ 1091us) server_negotiation.cc:299] Negotiation successful
0811 22:30:11.061190 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14170,"thread_start_us":14007,"threads_started":1}
I20260811 22:30:11.062769 30838 tablet_service.cc:1467] Processing CreateTablet for tablet 140553b999b641db8cf7c47b7e5eaf96 (DEFAULT_TABLE table=test-table [id=fd0c8aaafdaf4e368e9aa977e33a7ae6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:30:11.064036 30838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 140553b999b641db8cf7c47b7e5eaf96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:30:11.064602 30328 tablet_service.cc:1467] Processing CreateTablet for tablet 140553b999b641db8cf7c47b7e5eaf96 (DEFAULT_TABLE table=test-table [id=fd0c8aaafdaf4e368e9aa977e33a7ae6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:30:11.066067 30328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 140553b999b641db8cf7c47b7e5eaf96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:30:11.065796 30583 tablet_service.cc:1467] Processing CreateTablet for tablet 140553b999b641db8cf7c47b7e5eaf96 (DEFAULT_TABLE table=test-table [id=fd0c8aaafdaf4e368e9aa977e33a7ae6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:30:11.067080 30583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 140553b999b641db8cf7c47b7e5eaf96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:30:11.075526 30988 tablet_bootstrap.cc:492] T 140553b999b641db8cf7c47b7e5eaf96 P bbbefaf172cb424da1e736876b03cf89: Bootstrap starting.
I20260811 22:30:11.078609 30988 tablet_bootstrap.cc:654] T 140553b999b641db8cf7c47b7e5eaf96 P bbbefaf172cb424da1e736876b03cf89: Neither blocks nor log segments found. Creating new log.
I20260811 22:30:11.079684 30988 log.cc:824] T 140553b999b641db8cf7c47b7e5eaf96 P bbbefaf172cb424da1e736876b03cf89: Log is configured to *not* fsync() on all Append() calls
I20260811 22:30:11.080022 30989 tablet_bootstrap.cc:492] T 140553b999b641db8cf7c47b7e5eaf96 P dcd5e96211a04b37aa250ea0115bf45d: Bootstrap starting.
I20260811 22:30:11.084494 30989 tablet_bootstrap.cc:654] T 140553b999b641db8cf7c47b7e5eaf96 P dcd5e96211a04b37aa250ea0115bf45d: Neither blocks nor log segments found. Creating new log.
I20260811 22:30:11.085294 30990 tablet_bootstrap.cc:492] T 140553b999b641db8cf7c47b7e5eaf96 P 8259d78358174d538790ca8a6ad07798: Bootstrap starting.
I20260811 22:30:11.085831 30989 log.cc:824] T 140553b999b641db8cf7c47b7e5eaf96 P dcd5e96211a04b37aa250ea0115bf45d: Log is configured to *not* fsync() on all Append() calls
I20260811 22:30:11.088222 30988 tablet_bootstrap.cc:492] T 140553b999b641db8cf7c47b7e5eaf96 P bbbefaf172cb424da1e736876b03cf89: No bootstrap required, opened a new log
I20260811 22:30:11.088442 30990 tablet_bootstrap.cc:654] T 140553b999b641db8cf7c47b7e5eaf96 P 8259d78358174d538790ca8a6ad07798: Neither blocks nor log segments found. Creating new log.
I20260811 22:30:11.088497 30988 ts_tablet_manager.cc:1397] T 140553b999b641db8cf7c47b7e5eaf96 P bbbefaf172cb424da1e736876b03cf89: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.008s
I20260811 22:30:11.089422 30990 log.cc:824] T 140553b999b641db8cf7c47b7e5eaf96 P 8259d78358174d538790ca8a6ad07798: Log is configured to *not* fsync() on all Append() calls
I20260811 22:30:11.089998 30989 tablet_bootstrap.cc:492] T 140553b999b641db8cf7c47b7e5eaf96 P dcd5e96211a04b37aa250ea0115bf45d: No bootstrap required, opened a new log
I20260811 22:30:11.090278 30989 ts_tablet_manager.cc:1397] T 140553b999b641db8cf7c47b7e5eaf96 P dcd5e96211a04b37aa250ea0115bf45d: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260811 22:30:11.095992 30990 tablet_bootstrap.cc:492] T 140553b999b641db8cf7c47b7e5eaf96 P 8259d78358174d538790ca8a6ad07798: No bootstrap required, opened a new log
I20260811 22:30:11.096204 30990 ts_tablet_manager.cc:1397] T 140553b999b641db8cf7c47b7e5eaf96 P 8259d78358174d538790ca8a6ad07798: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20260811 22:30:11.098619 30988 raft_consensus.cc:348] T 140553b999b641db8cf7c47b7e5eaf96 P bbbefaf172cb424da1e736876b03cf89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8259d78358174d538790ca8a6ad07798" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37401 } } peers { permanent_uuid: "bbbefaf172cb424da1e736876b03cf89" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41169 } } peers { permanent_uuid: "dcd5e96211a04b37aa250ea0115bf45d" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 34143 } }
I20260811 22:30:11.099071 30988 raft_consensus.cc:374] T 140553b999b641db8cf7c47b7e5eaf96 P bbbefaf172cb424da1e736876b03cf89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:30:11.099169 30988 raft_consensus.cc:729] T 140553b999b641db8cf7c47b7e5eaf96 P bbbefaf172cb424da1e736876b03cf89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbbefaf172cb424da1e736876b03cf89, State: Initialized, Role: FOLLOWER
I20260811 22:30:11.099777 30988 consensus_queue.cc:260] T 140553b999b641db8cf7c47b7e5eaf96 P bbbefaf172cb424da1e736876b03cf89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8259d78358174d538790ca8a6ad07798" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37401 } } peers { permanent_uuid: "bbbefaf172cb424da1e736876b03cf89" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41169 } } peers { permanent_uuid: "dcd5e96211a04b37aa250ea0115bf45d" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 34143 } }
I20260811 22:30:11.100363 30989 raft_consensus.cc:348] T 140553b999b641db8cf7c47b7e5eaf96 P dcd5e96211a04b37aa250ea0115bf45d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8259d78358174d538790ca8a6ad07798" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37401 } } peers { permanent_uuid: "bbbefaf172cb424da1e736876b03cf89" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41169 } } peers { permanent_uuid: "dcd5e96211a04b37aa250ea0115bf45d" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 34143 } }
I20260811 22:30:11.100749 30989 raft_consensus.cc:374] T 140553b999b641db8cf7c47b7e5eaf96 P dcd5e96211a04b37aa250ea0115bf45d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:30:11.100849 30989 raft_consensus.cc:729] T 140553b999b641db8cf7c47b7e5eaf96 P dcd5e96211a04b37aa250ea0115bf45d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcd5e96211a04b37aa250ea0115bf45d, State: Initialized, Role: FOLLOWER
I20260811 22:30:11.100942 30952 heartbeater.cc:503] Master 127.29.106.62:37499 was elected leader, sending a full tablet report...
I20260811 22:30:11.101408 30989 consensus_queue.cc:260] T 140553b999b641db8cf7c47b7e5eaf96 P dcd5e96211a04b37aa250ea0115bf45d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8259d78358174d538790ca8a6ad07798" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37401 } } peers { permanent_uuid: "bbbefaf172cb424da1e736876b03cf89" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41169 } } peers { permanent_uuid: "dcd5e96211a04b37aa250ea0115bf45d" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 34143 } }
I20260811 22:30:11.102828 30988 ts_tablet_manager.cc:1428] T 140553b999b641db8cf7c47b7e5eaf96 P bbbefaf172cb424da1e736876b03cf89: Time spent starting tablet: real 0.014s user 0.006s sys 0.007s
I20260811 22:30:11.103113 30989 ts_tablet_manager.cc:1428] T 140553b999b641db8cf7c47b7e5eaf96 P dcd5e96211a04b37aa250ea0115bf45d: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
I20260811 22:30:11.105218 30999 raft_consensus.cc:482] T 140553b999b641db8cf7c47b7e5eaf96 P bbbefaf172cb424da1e736876b03cf89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:30:11.105461 30999 raft_consensus.cc:504] T 140553b999b641db8cf7c47b7e5eaf96 P bbbefaf172cb424da1e736876b03cf89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8259d78358174d538790ca8a6ad07798" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37401 } } peers { permanent_uuid: "bbbefaf172cb424da1e736876b03cf89" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41169 } } peers { permanent_uuid: "dcd5e96211a04b37aa250ea0115bf45d" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 34143 } }
I20260811 22:30:11.105396 30990 raft_consensus.cc:348] T 140553b999b641db8cf7c47b7e5eaf96 P 8259d78358174d538790ca8a6ad07798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8259d78358174d538790ca8a6ad07798" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37401 } } peers { permanent_uuid: "bbbefaf172cb424da1e736876b03cf89" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41169 } } peers { permanent_uuid: "dcd5e96211a04b37aa250ea0115bf45d" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 34143 } }
I20260811 22:30:11.105926 30990 raft_consensus.cc:374] T 140553b999b641db8cf7c47b7e5eaf96 P 8259d78358174d538790ca8a6ad07798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:30:11.106153 30990 raft_consensus.cc:729] T 140553b999b641db8cf7c47b7e5eaf96 P 8259d78358174d538790ca8a6ad07798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8259d78358174d538790ca8a6ad07798, State: Initialized, Role: FOLLOWER
I20260811 22:30:11.106851 30990 consensus_queue.cc:260] T 140553b999b641db8cf7c47b7e5eaf96 P 8259d78358174d538790ca8a6ad07798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8259d78358174d538790ca8a6ad07798" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37401 } } peers { permanent_uuid: "bbbefaf172cb424da1e736876b03cf89" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41169 } } peers { permanent_uuid: "dcd5e96211a04b37aa250ea0115bf45d" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 34143 } }
I20260811 22:30:11.108458 30999 leader_election.cc:288] T 140553b999b641db8cf7c47b7e5eaf96 P bbbefaf172cb424da1e736876b03cf89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8259d78358174d538790ca8a6ad07798 (127.29.106.2:37401), dcd5e96211a04b37aa250ea0115bf45d (127.29.106.1:34143)
I20260811 22:30:11.109225 30697 heartbeater.cc:503] Master 127.29.106.62:37499 was elected leader, sending a full tablet report...
I20260811 22:30:11.112308 31005 raft_consensus.cc:482] T 140553b999b641db8cf7c47b7e5eaf96 P 8259d78358174d538790ca8a6ad07798 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:30:11.112592 31005 raft_consensus.cc:504] T 140553b999b641db8cf7c47b7e5eaf96 P 8259d78358174d538790ca8a6ad07798 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8259d78358174d538790ca8a6ad07798" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37401 } } peers { permanent_uuid: "bbbefaf172cb424da1e736876b03cf89" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41169 } } peers { permanent_uuid: "dcd5e96211a04b37aa250ea0115bf45d" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 34143 } }
I20260811 22:30:11.114720 30990 ts_tablet_manager.cc:1428] T 140553b999b641db8cf7c47b7e5eaf96 P 8259d78358174d538790ca8a6ad07798: Time spent starting tablet: real 0.018s user 0.008s sys 0.003s
I20260811 22:30:11.117595 30960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:11.108251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:34143 (local address 127.29.106.3:52929)
0811 22:30:11.109082 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:11.109103 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:11.109225 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:11.109883 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:11.109889 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:11.109940 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:30:11.110132 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:11.110143 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:11.111810 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:11.111817 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:11.114512 (+ 2695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:11.114528 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:11.116734 (+ 2206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:11.116742 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:11.117007 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:11.117048 (+ 41us) client_negotiation.cc:769] Sending connection context
0811 22:30:11.117145 (+ 97us) client_negotiation.cc:240] Negotiation successful
0811 22:30:11.117294 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685}
I20260811 22:30:11.118294 30987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:11.107972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:52929 (local address 127.29.106.1:34143)
0811 22:30:11.108127 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:11.108132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:11.109456 (+ 1324us) server_negotiation.cc:407] Connection header received
0811 22:30:11.109534 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:11.109540 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:11.109626 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:11.109705 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:30:11.110295 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:11.111596 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:11.114732 (+ 3136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:11.116234 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:11.116416 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:11.116496 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:11.117870 (+ 1374us) server_negotiation.cc:299] Negotiation successful
0811 22:30:11.118036 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260811 22:30:11.119220 30358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "140553b999b641db8cf7c47b7e5eaf96" candidate_uuid: "bbbefaf172cb424da1e736876b03cf89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcd5e96211a04b37aa250ea0115bf45d" is_pre_election: true
I20260811 22:30:11.119812 30358 raft_consensus.cc:2454] T 140553b999b641db8cf7c47b7e5eaf96 P dcd5e96211a04b37aa250ea0115bf45d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbbefaf172cb424da1e736876b03cf89 in term 0.
I20260811 22:30:11.120565 30751 leader_election.cc:302] T 140553b999b641db8cf7c47b7e5eaf96 P bbbefaf172cb424da1e736876b03cf89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbbefaf172cb424da1e736876b03cf89, dcd5e96211a04b37aa250ea0115bf45d; no voters:
I20260811 22:30:11.121006 30999 raft_consensus.cc:2781] T 140553b999b641db8cf7c47b7e5eaf96 P bbbefaf172cb424da1e736876b03cf89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:30:11.121150 30999 raft_consensus.cc:482] T 140553b999b641db8cf7c47b7e5eaf96 P bbbefaf172cb424da1e736876b03cf89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:30:11.121243 30999 raft_consensus.cc:3037] T 140553b999b641db8cf7c47b7e5eaf96 P bbbefaf172cb424da1e736876b03cf89 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:30:11.124881 30999 raft_consensus.cc:504] T 140553b999b641db8cf7c47b7e5eaf96 P bbbefaf172cb424da1e736876b03cf89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8259d78358174d538790ca8a6ad07798" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37401 } } peers { permanent_uuid: "bbbefaf172cb424da1e736876b03cf89" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41169 } } peers { permanent_uuid: "dcd5e96211a04b37aa250ea0115bf45d" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 34143 } }
I20260811 22:30:11.125593 30999 leader_election.cc:288] T 140553b999b641db8cf7c47b7e5eaf96 P bbbefaf172cb424da1e736876b03cf89 [CANDIDATE]: Term 1 election: Requested vote from peers 8259d78358174d538790ca8a6ad07798 (127.29.106.2:37401), dcd5e96211a04b37aa250ea0115bf45d (127.29.106.1:34143)
I20260811 22:30:11.126245 30358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "140553b999b641db8cf7c47b7e5eaf96" candidate_uuid: "bbbefaf172cb424da1e736876b03cf89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcd5e96211a04b37aa250ea0115bf45d"
I20260811 22:30:11.126457 30358 raft_consensus.cc:3037] T 140553b999b641db8cf7c47b7e5eaf96 P dcd5e96211a04b37aa250ea0115bf45d [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:30:11.127923 30983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:11.107850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:56177 (local address 127.29.106.2:37401)
0811 22:30:11.108052 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:11.108058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:11.118988 (+ 10930us) server_negotiation.cc:407] Connection header received
0811 22:30:11.119336 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:11.119343 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:11.119419 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:11.119481 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:30:11.120129 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:11.121292 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:11.125211 (+ 3919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:11.126618 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:11.126759 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:11.126835 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:11.127479 (+ 644us) server_negotiation.cc:299] Negotiation successful
0811 22:30:11.127631 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260811 22:30:11.128633 30960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:11.108295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.2:37401 (local address 127.29.106.3:56177)
0811 22:30:11.118898 (+ 10603us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:11.118917 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:11.119164 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:11.119663 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:11.119668 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:11.119696 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:30:11.119967 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:11.119978 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:11.121519 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:11.121525 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:11.124964 (+ 3439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:11.124982 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:11.127027 (+ 2045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:11.127033 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:11.127285 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:11.127322 (+ 37us) client_negotiation.cc:769] Sending connection context
0811 22:30:11.128148 (+ 826us) client_negotiation.cc:240] Negotiation successful
0811 22:30:11.128300 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10481,"threads_started":1}
I20260811 22:30:11.129807 30358 raft_consensus.cc:2454] T 140553b999b641db8cf7c47b7e5eaf96 P dcd5e96211a04b37aa250ea0115bf45d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbbefaf172cb424da1e736876b03cf89 in term 1.
I20260811 22:30:11.130236 30751 leader_election.cc:302] T 140553b999b641db8cf7c47b7e5eaf96 P bbbefaf172cb424da1e736876b03cf89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbbefaf172cb424da1e736876b03cf89, dcd5e96211a04b37aa250ea0115bf45d; no voters:
I20260811 22:30:11.130519 30999 raft_consensus.cc:2781] T 140553b999b641db8cf7c47b7e5eaf96 P bbbefaf172cb424da1e736876b03cf89 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:30:11.130339 30620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "140553b999b641db8cf7c47b7e5eaf96" candidate_uuid: "bbbefaf172cb424da1e736876b03cf89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8259d78358174d538790ca8a6ad07798" is_pre_election: true
I20260811 22:30:11.130733 30617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "140553b999b641db8cf7c47b7e5eaf96" candidate_uuid: "bbbefaf172cb424da1e736876b03cf89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8259d78358174d538790ca8a6ad07798"
I20260811 22:30:11.130940 30620 raft_consensus.cc:2454] T 140553b999b641db8cf7c47b7e5eaf96 P 8259d78358174d538790ca8a6ad07798 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbbefaf172cb424da1e736876b03cf89 in term 0.
I20260811 22:30:11.131721 30999 raft_consensus.cc:686] T 140553b999b641db8cf7c47b7e5eaf96 P bbbefaf172cb424da1e736876b03cf89 [term 1 LEADER]: Becoming Leader. State: Replica: bbbefaf172cb424da1e736876b03cf89, State: Running, Role: LEADER
I20260811 22:30:11.132414 30999 consensus_queue.cc:237] T 140553b999b641db8cf7c47b7e5eaf96 P bbbefaf172cb424da1e736876b03cf89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8259d78358174d538790ca8a6ad07798" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37401 } } peers { permanent_uuid: "bbbefaf172cb424da1e736876b03cf89" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41169 } } peers { permanent_uuid: "dcd5e96211a04b37aa250ea0115bf45d" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 34143 } }
I20260811 22:30:11.133054 31005 leader_election.cc:288] T 140553b999b641db8cf7c47b7e5eaf96 P 8259d78358174d538790ca8a6ad07798 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbbefaf172cb424da1e736876b03cf89 (127.29.106.3:41169), dcd5e96211a04b37aa250ea0115bf45d (127.29.106.1:34143)
I20260811 22:30:11.137806 30071 catalog_manager.cc:5495] T 140553b999b641db8cf7c47b7e5eaf96 P bbbefaf172cb424da1e736876b03cf89 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbbefaf172cb424da1e736876b03cf89 (127.29.106.3). New cstate: current_term: 1 leader_uuid: "bbbefaf172cb424da1e736876b03cf89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8259d78358174d538790ca8a6ad07798" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbbefaf172cb424da1e736876b03cf89" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 41169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcd5e96211a04b37aa250ea0115bf45d" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 34143 } health_report { overall_health: UNKNOWN } } }
I20260811 22:30:11.140918 30985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:11.129462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:51113 (local address 127.29.106.3:41169)
0811 22:30:11.129630 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:11.129636 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:11.133947 (+ 4311us) server_negotiation.cc:407] Connection header received
0811 22:30:11.134182 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:11.134187 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:11.134249 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:11.134305 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:30:11.135150 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:11.136365 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:11.138738 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:11.139897 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:11.140050 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:11.140133 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:11.140588 (+ 455us) server_negotiation.cc:299] Negotiation successful
0811 22:30:11.140710 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260811 22:30:11.141016 31013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:11.114626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.3:41169 (local address 127.29.106.2:51113)
0811 22:30:11.133851 (+ 19225us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:11.133869 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:11.134046 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:11.134514 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:11.134520 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:11.134549 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:30:11.134969 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:11.134980 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:11.136509 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:11.136514 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:11.138562 (+ 2048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:11.138578 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:11.140149 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:11.140155 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:11.140323 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:11.140356 (+ 33us) client_negotiation.cc:769] Sending connection context
0811 22:30:11.140450 (+ 94us) client_negotiation.cc:240] Negotiation successful
0811 22:30:11.140575 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19101,"thread_start_us":23731,"threads_started":1}
I20260811 22:30:11.141783 30864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "140553b999b641db8cf7c47b7e5eaf96" candidate_uuid: "8259d78358174d538790ca8a6ad07798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbbefaf172cb424da1e736876b03cf89" is_pre_election: true
I20260811 22:30:11.145254 30987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:11.133745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:32891 (local address 127.29.106.1:34143)
0811 22:30:11.134334 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:11.134340 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:11.138584 (+ 4244us) server_negotiation.cc:407] Connection header received
0811 22:30:11.138651 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:11.138656 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:11.138720 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:11.138808 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:30:11.139458 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:11.140403 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:11.143655 (+ 3252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:11.144450 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:11.144602 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:11.144667 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:11.144966 (+ 299us) server_negotiation.cc:299] Negotiation successful
0811 22:30:11.145063 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464}
I20260811 22:30:11.145416 31011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:11.133314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:34143 (local address 127.29.106.2:32891)
0811 22:30:11.138136 (+ 4822us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:11.138155 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:11.138280 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:11.138839 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:11.138844 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:11.138870 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:30:11.139223 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:11.139234 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:11.140547 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:11.140553 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:11.143515 (+ 2962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:11.143528 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:11.144651 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:11.144657 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:11.144800 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:11.144837 (+ 37us) client_negotiation.cc:769] Sending connection context
0811 22:30:11.144919 (+ 82us) client_negotiation.cc:240] Negotiation successful
0811 22:30:11.145029 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4693,"thread_start_us":1034,"threads_started":1}
I20260811 22:30:11.146102 30358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "140553b999b641db8cf7c47b7e5eaf96" candidate_uuid: "8259d78358174d538790ca8a6ad07798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcd5e96211a04b37aa250ea0115bf45d" is_pre_election: true
I20260811 22:30:11.146409 30358 raft_consensus.cc:2379] T 140553b999b641db8cf7c47b7e5eaf96 P dcd5e96211a04b37aa250ea0115bf45d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8259d78358174d538790ca8a6ad07798 in current term 1: Already voted for candidate bbbefaf172cb424da1e736876b03cf89 in this term.
I20260811 22:30:11.147011 30481 leader_election.cc:302] T 140553b999b641db8cf7c47b7e5eaf96 P 8259d78358174d538790ca8a6ad07798 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8259d78358174d538790ca8a6ad07798; no voters: bbbefaf172cb424da1e736876b03cf89, dcd5e96211a04b37aa250ea0115bf45d
I20260811 22:30:11.147447 31005 raft_consensus.cc:3037] T 140553b999b641db8cf7c47b7e5eaf96 P 8259d78358174d538790ca8a6ad07798 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:30:11.149808 31005 raft_consensus.cc:2726] T 140553b999b641db8cf7c47b7e5eaf96 P 8259d78358174d538790ca8a6ad07798 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260811 22:30:11.154263 30985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:11.149150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45460 (local address 127.29.106.3:41169)
0811 22:30:11.149299 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:11.149305 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:11.149426 (+ 121us) server_negotiation.cc:407] Connection header received
0811 22:30:11.149635 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:11.149640 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:11.149710 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:11.149830 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:30:11.150196 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:11.150917 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:11.152267 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:11.152674 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:11.152779 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:11.153258 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:30:11.153708 (+ 450us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:30:11.153838 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:11.153945 (+ 107us) server_negotiation.cc:299] Negotiation successful
0811 22:30:11.154017 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260811 22:30:11.161702 30358 raft_consensus.cc:1261] T 140553b999b641db8cf7c47b7e5eaf96 P dcd5e96211a04b37aa250ea0115bf45d [term 1 FOLLOWER]: Refusing update from remote peer bbbefaf172cb424da1e736876b03cf89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:30:11.161736 30617 raft_consensus.cc:1261] T 140553b999b641db8cf7c47b7e5eaf96 P 8259d78358174d538790ca8a6ad07798 [term 1 FOLLOWER]: Refusing update from remote peer bbbefaf172cb424da1e736876b03cf89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:30:11.162500 31012 consensus_queue.cc:1035] T 140553b999b641db8cf7c47b7e5eaf96 P bbbefaf172cb424da1e736876b03cf89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8259d78358174d538790ca8a6ad07798" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 37401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:30:11.162858 30999 consensus_queue.cc:1035] T 140553b999b641db8cf7c47b7e5eaf96 P bbbefaf172cb424da1e736876b03cf89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcd5e96211a04b37aa250ea0115bf45d" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:30:11.174365 31016 mvcc.cc:205] Tried to move back new op lower bound from 7317452436109533184 to 7317452436002480128. Current Snapshot: MvccSnapshot[applied={T|T < 7317452436109533184}]
I20260811 22:30:11.177086 31018 mvcc.cc:205] Tried to move back new op lower bound from 7317452436109533184 to 7317452436002480128. Current Snapshot: MvccSnapshot[applied={T|T < 7317452436109533184}]
I20260811 22:30:11.188431 30071 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:30:11.188714 30071 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:30:11.192276 30071 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3a93f855414a41f589c8f94b6fa4f266: Sending DeleteTablet for 3 replicas of tablet 140553b999b641db8cf7c47b7e5eaf96
I20260811 22:30:11.193363 30583 tablet_service.cc:1514] Processing DeleteTablet for tablet 140553b999b641db8cf7c47b7e5eaf96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:30:11 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:56552
I20260811 22:30:11.193377 30838 tablet_service.cc:1514] Processing DeleteTablet for tablet 140553b999b641db8cf7c47b7e5eaf96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:30:11 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:45448
I20260811 22:30:11.193444 30328 tablet_service.cc:1514] Processing DeleteTablet for tablet 140553b999b641db8cf7c47b7e5eaf96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:30:11 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:36118
I20260811 22:30:11.196171 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 30214
I20260811 22:30:11.208293 31036 tablet_replica.cc:330] T 140553b999b641db8cf7c47b7e5eaf96 P dcd5e96211a04b37aa250ea0115bf45d: stopping tablet replica
I20260811 22:30:11.209022 31036 raft_consensus.cc:2229] T 140553b999b641db8cf7c47b7e5eaf96 P dcd5e96211a04b37aa250ea0115bf45d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:30:11.209512 31036 raft_consensus.cc:2258] T 140553b999b641db8cf7c47b7e5eaf96 P dcd5e96211a04b37aa250ea0115bf45d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:30:11.211774 31036 ts_tablet_manager.cc:1905] T 140553b999b641db8cf7c47b7e5eaf96 P dcd5e96211a04b37aa250ea0115bf45d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:30:11.220037 31036 ts_tablet_manager.cc:1918] T 140553b999b641db8cf7c47b7e5eaf96 P dcd5e96211a04b37aa250ea0115bf45d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:30:11.220192 31036 log.cc:1196] T 140553b999b641db8cf7c47b7e5eaf96 P dcd5e96211a04b37aa250ea0115bf45d: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-0/wal/wals/140553b999b641db8cf7c47b7e5eaf96
I20260811 22:30:11.220719 31036 ts_tablet_manager.cc:1939] T 140553b999b641db8cf7c47b7e5eaf96 P dcd5e96211a04b37aa250ea0115bf45d: Deleting consensus metadata
I20260811 22:30:11.222510 30042 catalog_manager.cc:4841] TS dcd5e96211a04b37aa250ea0115bf45d (127.29.106.1:34143): tablet 140553b999b641db8cf7c47b7e5eaf96 (table test-table [id=fd0c8aaafdaf4e368e9aa977e33a7ae6]) successfully deleted
I20260811 22:30:11.223212 31033 tablet_replica.cc:330] T 140553b999b641db8cf7c47b7e5eaf96 P bbbefaf172cb424da1e736876b03cf89: stopping tablet replica
I20260811 22:30:11.224283 31030 tablet_replica.cc:330] T 140553b999b641db8cf7c47b7e5eaf96 P 8259d78358174d538790ca8a6ad07798: stopping tablet replica
I20260811 22:30:11.224675 31030 raft_consensus.cc:2229] T 140553b999b641db8cf7c47b7e5eaf96 P 8259d78358174d538790ca8a6ad07798 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:30:11.224777 31033 raft_consensus.cc:2229] T 140553b999b641db8cf7c47b7e5eaf96 P bbbefaf172cb424da1e736876b03cf89 [term 1 LEADER]: Raft consensus shutting down.
I20260811 22:30:11.225148 31030 raft_consensus.cc:2258] T 140553b999b641db8cf7c47b7e5eaf96 P 8259d78358174d538790ca8a6ad07798 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:30:11.225405 31033 raft_consensus.cc:2258] T 140553b999b641db8cf7c47b7e5eaf96 P bbbefaf172cb424da1e736876b03cf89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:30:11.227372 31030 ts_tablet_manager.cc:1905] T 140553b999b641db8cf7c47b7e5eaf96 P 8259d78358174d538790ca8a6ad07798: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:30:11.228549 31033 ts_tablet_manager.cc:1905] T 140553b999b641db8cf7c47b7e5eaf96 P bbbefaf172cb424da1e736876b03cf89: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 11 22:30:11 dist-test-slave-gjrz krb5kdc[29984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.1@KRBTEST.COM
I20260811 22:30:11.234426 31030 ts_tablet_manager.cc:1918] T 140553b999b641db8cf7c47b7e5eaf96 P 8259d78358174d538790ca8a6ad07798: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:30:11.234586 31030 log.cc:1196] T 140553b999b641db8cf7c47b7e5eaf96 P 8259d78358174d538790ca8a6ad07798: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-1/wal/wals/140553b999b641db8cf7c47b7e5eaf96
I20260811 22:30:11.235101 31030 ts_tablet_manager.cc:1939] T 140553b999b641db8cf7c47b7e5eaf96 P 8259d78358174d538790ca8a6ad07798: Deleting consensus metadata
I20260811 22:30:11.235704 31033 ts_tablet_manager.cc:1918] T 140553b999b641db8cf7c47b7e5eaf96 P bbbefaf172cb424da1e736876b03cf89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:30:11.235850 31033 log.cc:1196] T 140553b999b641db8cf7c47b7e5eaf96 P bbbefaf172cb424da1e736876b03cf89: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786487105757681-30120-0/minicluster-data/ts-2/wal/wals/140553b999b641db8cf7c47b7e5eaf96
I20260811 22:30:11.236354 31033 ts_tablet_manager.cc:1939] T 140553b999b641db8cf7c47b7e5eaf96 P bbbefaf172cb424da1e736876b03cf89: Deleting consensus metadata
I20260811 22:30:11.236853 30049 catalog_manager.cc:4841] TS 8259d78358174d538790ca8a6ad07798 (127.29.106.2:37401): tablet 140553b999b641db8cf7c47b7e5eaf96 (table test-table [id=fd0c8aaafdaf4e368e9aa977e33a7ae6]) successfully deleted
I20260811 22:30:11.237809 30048 catalog_manager.cc:4841] TS bbbefaf172cb424da1e736876b03cf89 (127.29.106.3:41169): tablet 140553b999b641db8cf7c47b7e5eaf96 (table test-table [id=fd0c8aaafdaf4e368e9aa977e33a7ae6]) successfully deleted
I20260811 22:30:11.239696 30987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:11.197584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36122 (local address 127.29.106.1:34143)
0811 22:30:11.197763 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:11.197769 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:11.220639 (+ 22870us) server_negotiation.cc:407] Connection header received
0811 22:30:11.220989 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:11.220996 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:11.221084 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:11.221156 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:11.223825 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:11.224664 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:11.226625 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:11.227022 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:11.227131 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:11.232479 (+ 5348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:11.232494 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:11.232509 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:30:11.232548 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:11.236297 (+ 3749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:11.237366 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:11.237375 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:11.237379 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:11.237534 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:11.238079 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:11.238084 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:11.238088 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:11.238535 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:11.238843 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:11.239055 (+ 212us) server_negotiation.cc:299] Negotiation successful
0811 22:30:11.239344 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260811 22:30:11.240355 30439 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:30:11.330900 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 30214
I20260811 22:30:11.355041 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 30459
Aug 11 22:30:11 dist-test-slave-gjrz krb5kdc[29984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.2@KRBTEST.COM
I20260811 22:30:11.369959 30983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:11.355822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56556 (local address 127.29.106.2:37401)
0811 22:30:11.356007 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:11.356013 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:11.356082 (+ 69us) server_negotiation.cc:407] Connection header received
0811 22:30:11.356284 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:11.356290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:11.356361 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:11.356444 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:11.357762 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:11.358505 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:11.359972 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:11.360365 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:11.360477 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:11.364570 (+ 4093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:11.364585 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:11.364600 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:30:11.364635 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:11.367331 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:11.367984 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:11.367992 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:11.367994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:11.368119 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:11.368465 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:11.368469 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:11.368471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:11.368846 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:11.369121 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:11.369403 (+ 282us) server_negotiation.cc:299] Negotiation successful
0811 22:30:11.369623 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260811 22:30:11.370594 30690 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:30:11.460726 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 30459
I20260811 22:30:11.481611 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 30714
Aug 11 22:30:11 dist-test-slave-gjrz krb5kdc[29984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.3@KRBTEST.COM
I20260811 22:30:11.500499 30985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:11.482464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45470 (local address 127.29.106.3:41169)
0811 22:30:11.482662 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:11.482668 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:11.482686 (+ 18us) server_negotiation.cc:407] Connection header received
0811 22:30:11.482760 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:11.482764 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:11.482821 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:11.482918 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:11.484325 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:11.485063 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:11.486383 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:11.486828 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:11.486961 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:11.491701 (+ 4740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:11.491745 (+ 44us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:11.491771 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:30:11.491848 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:11.496563 (+ 4715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:11.497565 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:11.497575 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:11.497579 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:11.497744 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:11.498558 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:11.498564 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:11.498567 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:11.499134 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:11.499487 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:11.499887 (+ 400us) server_negotiation.cc:299] Negotiation successful
0811 22:30:11.500132 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260811 22:30:11.501250 30945 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:30:11.622233 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 30714
I20260811 22:30:11.645478 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 30011
I20260811 22:30:11.646596 30147 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:30:11.756131 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 30011
2026-08-11T22:30:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4365 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-11T22:30:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:30:11Z Disabled control of system clock
I20260811 22:30:11.846690 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:42541
--webserver_interface=127.29.106.62
--webserver_port=0
--builtin_ntp_servers=127.29.106.20:44551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:42541
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 22:30:12.175910 31050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:30:12.176199 31050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:30:12.176291 31050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:30:12.183444 31050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:30:12.183539 31050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:30:12.183583 31050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:30:12.183640 31050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:30:12.183686 31050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:30:12.192610 31050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:44551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:42541
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:42541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.106.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:30:12.193766 31050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:30:12.195506 31050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:30:12.206857 31070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:12.207182 31059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:12.206960 31061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:30:12.208197 31050 server_base.cc:1034] running on GCE node
I20260811 22:30:12.208847 31050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:30:12.210175 31050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:30:12.211354 31050 hybrid_clock.cc:648] HybridClock initialized: now 1786487412211315 us; error 32 us; skew 500 ppm
I20260811 22:30:12.211663 31050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 22:30:12.218569 31050 webserver.cc:460] Webserver started at http://127.29.106.62:34515/ using document root <none> and password file <none>
I20260811 22:30:12.219172 31050 fs_manager.cc:359] Metadata directory not provided
I20260811 22:30:12.219259 31050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:30:12.219579 31050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:30:12.222388 31050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "d2defc52470e47ce863e6b5d28468b14"
format_stamp: "Formatted at 2026-08-11 22:30:12 on dist-test-slave-gjrz"
server_key: "bcfd142fba5c2be95372b0e67edcc3f9"
server_key_iv: "dc42ca741cd5d9ee5112ba796efffef2"
server_key_version: "encryptionkey@0"
I20260811 22:30:12.223125 31050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "d2defc52470e47ce863e6b5d28468b14"
format_stamp: "Formatted at 2026-08-11 22:30:12 on dist-test-slave-gjrz"
server_key: "bcfd142fba5c2be95372b0e67edcc3f9"
server_key_iv: "dc42ca741cd5d9ee5112ba796efffef2"
server_key_version: "encryptionkey@0"
I20260811 22:30:12.227905 31050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260811 22:30:12.231125 31078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:12.232367 31050 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260811 22:30:12.232522 31050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "d2defc52470e47ce863e6b5d28468b14"
format_stamp: "Formatted at 2026-08-11 22:30:12 on dist-test-slave-gjrz"
server_key: "bcfd142fba5c2be95372b0e67edcc3f9"
server_key_iv: "dc42ca741cd5d9ee5112ba796efffef2"
server_key_version: "encryptionkey@0"
I20260811 22:30:12.232707 31050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:30:12.250095 31050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:30:12.250857 31050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:30:12.251132 31050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:30:12.268878 31050 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.62:42541
I20260811 22:30:12.268934 31167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.62:42541 every 8 connection(s)
I20260811 22:30:12.270081 31050 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:30:12.270926 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 31050
I20260811 22:30:12.271111 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:30:12.271689 30120 external_mini_cluster.cc:1428] Setting key 96d73e05907601c379589acc54f6e9d3
I20260811 22:30:12.274525 31168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:30:12.284233 31172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:12.273542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41664 (local address 127.29.106.62:42541)
0811 22:30:12.274455 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:12.274465 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:12.274502 (+ 37us) server_negotiation.cc:407] Connection header received
0811 22:30:12.276181 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:12.276202 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:12.276602 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:12.276964 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:12.277499 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:12.278787 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:12.280206 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:12.280836 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:12.281007 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:12.282087 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:12.282097 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:12.282110 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:30:12.282648 (+ 538us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:12.282722 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:30:12.282747 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:12.282858 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:12.283090 (+ 232us) server_negotiation.cc:299] Negotiation successful
0811 22:30:12.283396 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":329,"threads_started":1}
I20260811 22:30:12.284858 31168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2defc52470e47ce863e6b5d28468b14: Bootstrap starting.
I20260811 22:30:12.287952 31168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2defc52470e47ce863e6b5d28468b14: Neither blocks nor log segments found. Creating new log.
I20260811 22:30:12.288904 31168 log.cc:824] T 00000000000000000000000000000000 P d2defc52470e47ce863e6b5d28468b14: Log is configured to *not* fsync() on all Append() calls
I20260811 22:30:12.291627 31168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2defc52470e47ce863e6b5d28468b14: No bootstrap required, opened a new log
I20260811 22:30:12.298599 31168 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2defc52470e47ce863e6b5d28468b14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2defc52470e47ce863e6b5d28468b14" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 42541 } }
I20260811 22:30:12.298916 31168 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2defc52470e47ce863e6b5d28468b14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:30:12.298993 31168 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2defc52470e47ce863e6b5d28468b14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2defc52470e47ce863e6b5d28468b14, State: Initialized, Role: FOLLOWER
I20260811 22:30:12.299482 31168 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2defc52470e47ce863e6b5d28468b14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2defc52470e47ce863e6b5d28468b14" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 42541 } }
I20260811 22:30:12.299698 31168 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2defc52470e47ce863e6b5d28468b14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:30:12.299791 31168 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2defc52470e47ce863e6b5d28468b14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:30:12.299943 31168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2defc52470e47ce863e6b5d28468b14 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:30:12.301856 31168 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2defc52470e47ce863e6b5d28468b14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2defc52470e47ce863e6b5d28468b14" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 42541 } }
I20260811 22:30:12.302313 31168 leader_election.cc:302] T 00000000000000000000000000000000 P d2defc52470e47ce863e6b5d28468b14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2defc52470e47ce863e6b5d28468b14; no voters:
I20260811 22:30:12.302819 31168 leader_election.cc:288] T 00000000000000000000000000000000 P d2defc52470e47ce863e6b5d28468b14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:30:12.302953 31175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2defc52470e47ce863e6b5d28468b14 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:30:12.304037 31175 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2defc52470e47ce863e6b5d28468b14 [term 1 LEADER]: Becoming Leader. State: Replica: d2defc52470e47ce863e6b5d28468b14, State: Running, Role: LEADER
I20260811 22:30:12.304666 31175 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2defc52470e47ce863e6b5d28468b14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2defc52470e47ce863e6b5d28468b14" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 42541 } }
I20260811 22:30:12.304924 31168 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2defc52470e47ce863e6b5d28468b14 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:30:12.308207 31177 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2defc52470e47ce863e6b5d28468b14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2defc52470e47ce863e6b5d28468b14. Latest consensus state: current_term: 1 leader_uuid: "d2defc52470e47ce863e6b5d28468b14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2defc52470e47ce863e6b5d28468b14" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 42541 } } }
I20260811 22:30:12.308873 31177 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2defc52470e47ce863e6b5d28468b14 [sys.catalog]: This master's current role is: LEADER
I20260811 22:30:12.308279 31176 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2defc52470e47ce863e6b5d28468b14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2defc52470e47ce863e6b5d28468b14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2defc52470e47ce863e6b5d28468b14" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 42541 } } }
I20260811 22:30:12.309196 31176 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2defc52470e47ce863e6b5d28468b14 [sys.catalog]: This master's current role is: LEADER
I20260811 22:30:12.310352 31193 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:30:12.314476 31193 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:30:12.322291 31193 catalog_manager.cc:1347] Generated new cluster ID: 6962f6f05cba4410852d7098bf53ced5
I20260811 22:30:12.322381 31193 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:30:12.343075 31193 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:30:12.344336 31193 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:30:12.355182 31193 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2defc52470e47ce863e6b5d28468b14: Generated new TSK 0
I20260811 22:30:12.355958 31193 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:30:12.366245 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.1:0
--local_ip_for_outbound_sockets=127.29.106.1
--webserver_interface=127.29.106.1
--webserver_port=0
--tserver_master_addrs=127.29.106.62:42541
--builtin_ntp_servers=127.29.106.20:44551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 22:30:12.698159 31204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:30:12.698428 31204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:30:12.698541 31204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:30:12.706018 31204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:30:12.706123 31204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:30:12.706266 31204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.1
I20260811 22:30:12.715601 31204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:44551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.106.1
--webserver_port=0
--tserver_master_addrs=127.29.106.62:42541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.1
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:30:12.716886 31204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:30:12.718638 31204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:30:12.731148 31217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:12.731125 31212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:12.731489 31223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:30:12.732585 31204 server_base.cc:1034] running on GCE node
I20260811 22:30:12.733189 31204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:30:12.734042 31204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:30:12.735247 31204 hybrid_clock.cc:648] HybridClock initialized: now 1786487412735198 us; error 40 us; skew 500 ppm
I20260811 22:30:12.735571 31204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 22:30:12.737746 31204 webserver.cc:460] Webserver started at http://127.29.106.1:33111/ using document root <none> and password file <none>
I20260811 22:30:12.738369 31204 fs_manager.cc:359] Metadata directory not provided
I20260811 22:30:12.738461 31204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:30:12.738750 31204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:30:12.741685 31204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-0/data/instance:
uuid: "1096bacf18334b1eba94d3cd4835b68d"
format_stamp: "Formatted at 2026-08-11 22:30:12 on dist-test-slave-gjrz"
server_key: "a9a605adf5ccbfc6dbd1f2254ad57ddf"
server_key_iv: "c7a8047ec43977c27e6e9fff362e9c0b"
server_key_version: "encryptionkey@0"
I20260811 22:30:12.742441 31204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance:
uuid: "1096bacf18334b1eba94d3cd4835b68d"
format_stamp: "Formatted at 2026-08-11 22:30:12 on dist-test-slave-gjrz"
server_key: "a9a605adf5ccbfc6dbd1f2254ad57ddf"
server_key_iv: "c7a8047ec43977c27e6e9fff362e9c0b"
server_key_version: "encryptionkey@0"
I20260811 22:30:12.747164 31204 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260811 22:30:12.750365 31235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:12.751587 31204 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:30:12.751730 31204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-0/wal
uuid: "1096bacf18334b1eba94d3cd4835b68d"
format_stamp: "Formatted at 2026-08-11 22:30:12 on dist-test-slave-gjrz"
server_key: "a9a605adf5ccbfc6dbd1f2254ad57ddf"
server_key_iv: "c7a8047ec43977c27e6e9fff362e9c0b"
server_key_version: "encryptionkey@0"
I20260811 22:30:12.751912 31204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:30:12.769976 31204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:30:12.770675 31204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:30:12.770942 31204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:30:12.771890 31204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:30:12.773211 31204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:30:12.773306 31204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:12.773406 31204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:30:12.773483 31204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:12.803253 31204 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.1:36713
I20260811 22:30:12.803342 31434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.1:36713 every 8 connection(s)
I20260811 22:30:12.804447 31204 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
I20260811 22:30:12.814396 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 31204
I20260811 22:30:12.814555 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance
I20260811 22:30:12.814980 30120 external_mini_cluster.cc:1428] Setting key 838c2f87dfe695ecf1fbd80f60ff57f5
I20260811 22:30:12.816565 31437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:12.806792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:33459 (local address 127.29.106.62:42541)
0811 22:30:12.807218 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:12.807223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:12.808417 (+ 1194us) server_negotiation.cc:407] Connection header received
0811 22:30:12.810336 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:12.810343 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:12.810423 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:12.810519 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:12.811570 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:12.812291 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:12.813823 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:12.814257 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:12.814368 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:12.815710 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:12.815715 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:12.815719 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:30:12.815735 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:12.815789 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:30:12.815801 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:12.815874 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:12.816105 (+ 231us) server_negotiation.cc:299] Negotiation successful
0811 22:30:12.816203 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":211,"threads_started":1}
I20260811 22:30:12.817590 31438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:12.807367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42541 (local address 127.29.106.1:33459)
0811 22:30:12.808270 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:12.808306 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:12.810076 (+ 1770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:12.810752 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:12.810760 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:12.810839 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:30:12.811408 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:12.811427 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:12.812449 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:12.812453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:12.813685 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:12.813698 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:12.814472 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:12.814480 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:12.814740 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:12.815395 (+ 655us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 22:30:12.815398 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:30:12.815441 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 22:30:12.815516 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:30:12.815890 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:30:12.815903 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:30:12.815911 (+ 8us) client_negotiation.cc:769] Sending connection context
0811 22:30:12.816049 (+ 138us) client_negotiation.cc:240] Negotiation successful
0811 22:30:12.816410 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":296,"threads_started":1}
I20260811 22:30:12.817945 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.2:0
--local_ip_for_outbound_sockets=127.29.106.2
--webserver_interface=127.29.106.2
--webserver_port=0
--tserver_master_addrs=127.29.106.62:42541
--builtin_ntp_servers=127.29.106.20:44551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260811 22:30:12.818941 31435 heartbeater.cc:347] Connected to a master server at 127.29.106.62:42541
I20260811 22:30:12.819247 31435 heartbeater.cc:464] Registering TS with master...
I20260811 22:30:12.820109 31435 heartbeater.cc:511] Master 127.29.106.62:42541 requested a full tablet report, sending...
I20260811 22:30:12.822134 31107 ts_manager.cc:194] Registered new tserver with Master: 1096bacf18334b1eba94d3cd4835b68d (127.29.106.1:36713)
I20260811 22:30:12.823630 31107 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.106.1:33459
W20260811 22:30:13.142632 31439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:30:13.142879 31439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:30:13.142971 31439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:30:13.149845 31439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:30:13.149930 31439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:30:13.150036 31439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.2
I20260811 22:30:13.158699 31439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:44551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.106.2
--webserver_port=0
--tserver_master_addrs=127.29.106.62:42541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.2
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:30:13.159878 31439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:30:13.161693 31439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:30:13.174062 31445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:13.174172 31446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:13.174217 31450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:30:13.174885 31439 server_base.cc:1034] running on GCE node
I20260811 22:30:13.175478 31439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:30:13.176355 31439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:30:13.177567 31439 hybrid_clock.cc:648] HybridClock initialized: now 1786487413177521 us; error 36 us; skew 500 ppm
I20260811 22:30:13.177861 31439 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 22:30:13.180100 31439 webserver.cc:460] Webserver started at http://127.29.106.2:32893/ using document root <none> and password file <none>
I20260811 22:30:13.180763 31439 fs_manager.cc:359] Metadata directory not provided
I20260811 22:30:13.180852 31439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:30:13.181173 31439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:30:13.183981 31439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-1/data/instance:
uuid: "909609d27239488682586693d2e1cdc1"
format_stamp: "Formatted at 2026-08-11 22:30:13 on dist-test-slave-gjrz"
server_key: "5dbb5fe890fc28598e9efd91d0cdbd9e"
server_key_iv: "86d19e1f52f72c6757b07156ed6588c9"
server_key_version: "encryptionkey@0"
I20260811 22:30:13.184736 31439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance:
uuid: "909609d27239488682586693d2e1cdc1"
format_stamp: "Formatted at 2026-08-11 22:30:13 on dist-test-slave-gjrz"
server_key: "5dbb5fe890fc28598e9efd91d0cdbd9e"
server_key_iv: "86d19e1f52f72c6757b07156ed6588c9"
server_key_version: "encryptionkey@0"
I20260811 22:30:13.189570 31439 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 22:30:13.192847 31461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:13.194023 31439 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 22:30:13.194180 31439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-1/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-1/wal
uuid: "909609d27239488682586693d2e1cdc1"
format_stamp: "Formatted at 2026-08-11 22:30:13 on dist-test-slave-gjrz"
server_key: "5dbb5fe890fc28598e9efd91d0cdbd9e"
server_key_iv: "86d19e1f52f72c6757b07156ed6588c9"
server_key_version: "encryptionkey@0"
I20260811 22:30:13.194366 31439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:30:13.210592 31439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:30:13.211374 31439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:30:13.211655 31439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:30:13.212555 31439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:30:13.213912 31439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:30:13.214013 31439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:13.214124 31439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:30:13.214197 31439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:13.244400 31439 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.2:39595
I20260811 22:30:13.244451 31661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.2:39595 every 8 connection(s)
I20260811 22:30:13.245663 31439 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
I20260811 22:30:13.246222 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 31439
I20260811 22:30:13.246377 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance
I20260811 22:30:13.246937 30120 external_mini_cluster.cc:1428] Setting key 779175c2bad60273a4b4d7bbfae797b4
I20260811 22:30:13.250049 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.3:0
--local_ip_for_outbound_sockets=127.29.106.3
--webserver_interface=127.29.106.3
--webserver_port=0
--tserver_master_addrs=127.29.106.62:42541
--builtin_ntp_servers=127.29.106.20:44551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260811 22:30:13.257982 31437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:13.248334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:41795 (local address 127.29.106.62:42541)
0811 22:30:13.248529 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:13.248545 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:13.250120 (+ 1575us) server_negotiation.cc:407] Connection header received
0811 22:30:13.251937 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:13.251942 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:13.252018 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:13.252109 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:13.253149 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:13.253845 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:13.255421 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:13.255826 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:13.255951 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:13.257214 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:13.257219 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:13.257223 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:30:13.257240 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:13.257270 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:30:13.257282 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:13.257353 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:13.257560 (+ 207us) server_negotiation.cc:299] Negotiation successful
0811 22:30:13.257659 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260811 22:30:13.258916 31666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:13.248935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42541 (local address 127.29.106.2:41795)
0811 22:30:13.249968 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:13.250004 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:13.251688 (+ 1684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:13.252355 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:13.252363 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:13.252445 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:30:13.252991 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:13.253007 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:13.254001 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:13.254004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:13.255282 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:13.255300 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:13.256048 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:13.256056 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:13.256276 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:13.256929 (+ 653us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 22:30:13.256932 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:30:13.256968 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 22:30:13.257038 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:30:13.257367 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:30:13.257379 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:30:13.257387 (+ 8us) client_negotiation.cc:769] Sending connection context
0811 22:30:13.257534 (+ 147us) client_negotiation.cc:240] Negotiation successful
0811 22:30:13.257922 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":308,"threads_started":1}
I20260811 22:30:13.260284 31663 heartbeater.cc:347] Connected to a master server at 127.29.106.62:42541
I20260811 22:30:13.260551 31663 heartbeater.cc:464] Registering TS with master...
I20260811 22:30:13.261246 31663 heartbeater.cc:511] Master 127.29.106.62:42541 requested a full tablet report, sending...
I20260811 22:30:13.262559 31107 ts_manager.cc:194] Registered new tserver with Master: 909609d27239488682586693d2e1cdc1 (127.29.106.2:39595)
I20260811 22:30:13.263370 31107 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.106.2:41795
W20260811 22:30:13.575387 31667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:30:13.575655 31667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:30:13.575785 31667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:30:13.582897 31667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:30:13.582998 31667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:30:13.583150 31667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.3
I20260811 22:30:13.591897 31667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:44551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.106.3
--webserver_port=0
--tserver_master_addrs=127.29.106.62:42541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.3
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:30:13.593055 31667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:30:13.594749 31667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:30:13.607530 31681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:13.607482 31682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:13.607622 31685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:30:13.608582 31667 server_base.cc:1034] running on GCE node
I20260811 22:30:13.609122 31667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:30:13.609952 31667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:30:13.611140 31667 hybrid_clock.cc:648] HybridClock initialized: now 1786487413611097 us; error 32 us; skew 500 ppm
I20260811 22:30:13.611447 31667 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 22:30:13.613544 31667 webserver.cc:460] Webserver started at http://127.29.106.3:35527/ using document root <none> and password file <none>
I20260811 22:30:13.614156 31667 fs_manager.cc:359] Metadata directory not provided
I20260811 22:30:13.614252 31667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:30:13.614542 31667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:30:13.617481 31667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-2/data/instance:
uuid: "c0551f1684494eb89f0a382480942106"
format_stamp: "Formatted at 2026-08-11 22:30:13 on dist-test-slave-gjrz"
server_key: "12fbaced314f925fa1acc64a5b7bf54a"
server_key_iv: "9141a0923a625d5afd303e04f3e34817"
server_key_version: "encryptionkey@0"
I20260811 22:30:13.618263 31667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance:
uuid: "c0551f1684494eb89f0a382480942106"
format_stamp: "Formatted at 2026-08-11 22:30:13 on dist-test-slave-gjrz"
server_key: "12fbaced314f925fa1acc64a5b7bf54a"
server_key_iv: "9141a0923a625d5afd303e04f3e34817"
server_key_version: "encryptionkey@0"
I20260811 22:30:13.622963 31667 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260811 22:30:13.626214 31695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:13.627455 31667 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260811 22:30:13.627594 31667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-2/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-2/wal
uuid: "c0551f1684494eb89f0a382480942106"
format_stamp: "Formatted at 2026-08-11 22:30:13 on dist-test-slave-gjrz"
server_key: "12fbaced314f925fa1acc64a5b7bf54a"
server_key_iv: "9141a0923a625d5afd303e04f3e34817"
server_key_version: "encryptionkey@0"
I20260811 22:30:13.627776 31667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:30:13.645340 31667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:30:13.646020 31667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:30:13.646291 31667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:30:13.647154 31667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:30:13.648479 31667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:30:13.648578 31667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:13.648690 31667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:30:13.648761 31667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:13.678603 31667 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.3:42515
I20260811 22:30:13.678663 31911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.3:42515 every 8 connection(s)
I20260811 22:30:13.679922 31667 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
I20260811 22:30:13.689299 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 31667
I20260811 22:30:13.689458 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance
I20260811 22:30:13.689987 30120 external_mini_cluster.cc:1428] Setting key 38d186c71b65b8758b86ec607151df60
I20260811 22:30:13.692144 31437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:13.682385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:41815 (local address 127.29.106.62:42541)
0811 22:30:13.682557 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:13.682561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:13.683933 (+ 1372us) server_negotiation.cc:407] Connection header received
0811 22:30:13.685630 (+ 1697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:13.685635 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:13.685704 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:13.685792 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:13.686759 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:13.687586 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:13.689193 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:13.689696 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:13.689802 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:13.691281 (+ 1479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:13.691288 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:13.691293 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:30:13.691315 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:13.691360 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:30:13.691378 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:13.691481 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:13.691698 (+ 217us) server_negotiation.cc:299] Negotiation successful
0811 22:30:13.691791 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260811 22:30:13.693156 31916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:13.682886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:42541 (local address 127.29.106.3:41815)
0811 22:30:13.683779 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:13.683822 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:13.685379 (+ 1557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:13.685974 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:13.685981 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:13.686075 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:30:13.686602 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:13.686618 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:13.687714 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:13.687717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:13.689006 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:13.689022 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:13.689926 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:13.689934 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:13.690242 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:13.690952 (+ 710us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 22:30:13.690960 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:30:13.691000 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 22:30:13.691086 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:30:13.691483 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:30:13.691501 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:30:13.691507 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 22:30:13.691677 (+ 170us) client_negotiation.cc:240] Negotiation successful
0811 22:30:13.692007 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":323,"threads_started":1}
I20260811 22:30:13.694334 31912 heartbeater.cc:347] Connected to a master server at 127.29.106.62:42541
I20260811 22:30:13.694578 31912 heartbeater.cc:464] Registering TS with master...
I20260811 22:30:13.695293 31912 heartbeater.cc:511] Master 127.29.106.62:42541 requested a full tablet report, sending...
I20260811 22:30:13.696516 31107 ts_manager.cc:194] Registered new tserver with Master: c0551f1684494eb89f0a382480942106 (127.29.106.3:42515)
I20260811 22:30:13.697324 31107 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.106.3:41815
I20260811 22:30:13.704962 30120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 22:30:13.714421 31437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:13.708903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41668 (local address 127.29.106.62:42541)
0811 22:30:13.709052 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:13.709056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:13.709485 (+ 429us) server_negotiation.cc:407] Connection header received
0811 22:30:13.709643 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:13.709648 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:13.709729 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:13.709806 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:13.710270 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:13.710952 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:13.712308 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:13.712705 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:13.712812 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:13.713784 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:13.713789 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:13.713792 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:30:13.713808 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:13.713837 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:30:13.713849 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:13.713920 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:13.714066 (+ 146us) server_negotiation.cc:299] Negotiation successful
0811 22:30:13.714154 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260811 22:30:13.717828 31107 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41668:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:30:13.724328 31107 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:30:13.760063 31943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:13.736330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36074 (local address 127.29.106.2:39595)
0811 22:30:13.746583 (+ 10253us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:13.746602 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:13.750501 (+ 3899us) server_negotiation.cc:407] Connection header received
0811 22:30:13.750973 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:13.750983 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:13.751242 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:13.751442 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:30:13.752560 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:13.754256 (+ 1696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:13.757089 (+ 2833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:13.758463 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:13.758579 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:13.758850 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:13.759568 (+ 718us) server_negotiation.cc:299] Negotiation successful
0811 22:30:13.759733 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10072,"thread_start_us":9936,"threads_started":1}
I20260811 22:30:13.760813 31932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:13.735609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.2:39595 (local address 127.0.0.1:36074)
0811 22:30:13.750391 (+ 14782us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:13.750411 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:13.750760 (+ 349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:13.751873 (+ 1113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:13.751881 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:13.751915 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:30:13.752334 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:13.752347 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:13.754463 (+ 2116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:13.754468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:13.756894 (+ 2426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:13.756909 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:13.759036 (+ 2127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:13.759044 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:13.759257 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:13.759336 (+ 79us) client_negotiation.cc:769] Sending connection context
0811 22:30:13.760321 (+ 985us) client_negotiation.cc:240] Negotiation successful
0811 22:30:13.760521 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14650,"thread_start_us":19088,"threads_started":1}
I20260811 22:30:13.762087 31941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:13.735923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:36713 (local address 127.0.0.1:56576)
0811 22:30:13.755005 (+ 19082us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:13.755027 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:13.755162 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:13.755752 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:13.755756 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:13.755784 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:30:13.756247 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:13.756257 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:13.757919 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:13.757925 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:13.759636 (+ 1711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:13.759649 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:13.761322 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:13.761329 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:13.761508 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:13.761539 (+ 31us) client_negotiation.cc:769] Sending connection context
0811 22:30:13.761627 (+ 88us) client_negotiation.cc:240] Negotiation successful
0811 22:30:13.761746 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18930,"mutex_wait_us":59,"thread_start_us":13931,"threads_started":1}
I20260811 22:30:13.762827 31942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:13.735257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.3:42515 (local address 127.0.0.1:54494)
0811 22:30:13.750306 (+ 15049us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:13.750352 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:13.750528 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:13.755324 (+ 4796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:13.755328 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:13.755351 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:30:13.756157 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:13.756170 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:13.757957 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:13.757962 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:13.760117 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:13.760132 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:13.762157 (+ 2025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:13.762164 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:13.762333 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:13.762364 (+ 31us) client_negotiation.cc:769] Sending connection context
0811 22:30:13.762437 (+ 73us) client_negotiation.cc:240] Negotiation successful
0811 22:30:13.762549 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14732,"thread_start_us":15252,"threads_started":1}
I20260811 22:30:13.762857 31944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:13.736638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56576 (local address 127.29.106.1:36713)
0811 22:30:13.754030 (+ 17392us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:13.754038 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:13.755156 (+ 1118us) server_negotiation.cc:407] Connection header received
0811 22:30:13.755325 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:13.755333 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:13.755548 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:13.755747 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:30:13.756417 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:13.757785 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:13.759814 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:13.760839 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:13.760969 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:13.761199 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:13.762374 (+ 1175us) server_negotiation.cc:299] Negotiation successful
0811 22:30:13.762551 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17206,"thread_start_us":17400,"threads_started":1}
I20260811 22:30:13.763530 31938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:13.735537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54494 (local address 127.29.106.3:42515)
0811 22:30:13.754648 (+ 19111us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:13.754656 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:13.754692 (+ 36us) server_negotiation.cc:407] Connection header received
0811 22:30:13.754787 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:13.754794 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:13.755051 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:13.755327 (+ 276us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:30:13.756355 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:13.757799 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:13.760324 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:13.761629 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:13.761743 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:13.762002 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:13.763087 (+ 1085us) server_negotiation.cc:299] Negotiation successful
0811 22:30:13.763222 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18960,"thread_start_us":18822,"threads_started":1}
I20260811 22:30:13.765149 31552 tablet_service.cc:1467] Processing CreateTablet for tablet 782f62eb19c14cb1bacdf41fc5855994 (DEFAULT_TABLE table=test-table [id=774432f14f6543ff8f46b05e8bd56727]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:30:13.766106 31317 tablet_service.cc:1467] Processing CreateTablet for tablet 782f62eb19c14cb1bacdf41fc5855994 (DEFAULT_TABLE table=test-table [id=774432f14f6543ff8f46b05e8bd56727]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:30:13.767292 31552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 782f62eb19c14cb1bacdf41fc5855994. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:30:13.767303 31317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 782f62eb19c14cb1bacdf41fc5855994. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:30:13.766947 31793 tablet_service.cc:1467] Processing CreateTablet for tablet 782f62eb19c14cb1bacdf41fc5855994 (DEFAULT_TABLE table=test-table [id=774432f14f6543ff8f46b05e8bd56727]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:30:13.768472 31793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 782f62eb19c14cb1bacdf41fc5855994. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:30:13.777985 31945 tablet_bootstrap.cc:492] T 782f62eb19c14cb1bacdf41fc5855994 P 1096bacf18334b1eba94d3cd4835b68d: Bootstrap starting.
I20260811 22:30:13.781509 31945 tablet_bootstrap.cc:654] T 782f62eb19c14cb1bacdf41fc5855994 P 1096bacf18334b1eba94d3cd4835b68d: Neither blocks nor log segments found. Creating new log.
I20260811 22:30:13.781705 31946 tablet_bootstrap.cc:492] T 782f62eb19c14cb1bacdf41fc5855994 P 909609d27239488682586693d2e1cdc1: Bootstrap starting.
I20260811 22:30:13.782828 31945 log.cc:824] T 782f62eb19c14cb1bacdf41fc5855994 P 1096bacf18334b1eba94d3cd4835b68d: Log is configured to *not* fsync() on all Append() calls
I20260811 22:30:13.784204 31950 tablet_bootstrap.cc:492] T 782f62eb19c14cb1bacdf41fc5855994 P c0551f1684494eb89f0a382480942106: Bootstrap starting.
I20260811 22:30:13.785156 31946 tablet_bootstrap.cc:654] T 782f62eb19c14cb1bacdf41fc5855994 P 909609d27239488682586693d2e1cdc1: Neither blocks nor log segments found. Creating new log.
I20260811 22:30:13.785713 31945 tablet_bootstrap.cc:492] T 782f62eb19c14cb1bacdf41fc5855994 P 1096bacf18334b1eba94d3cd4835b68d: No bootstrap required, opened a new log
I20260811 22:30:13.785948 31945 ts_tablet_manager.cc:1397] T 782f62eb19c14cb1bacdf41fc5855994 P 1096bacf18334b1eba94d3cd4835b68d: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.002s
I20260811 22:30:13.786319 31946 log.cc:824] T 782f62eb19c14cb1bacdf41fc5855994 P 909609d27239488682586693d2e1cdc1: Log is configured to *not* fsync() on all Append() calls
I20260811 22:30:13.787503 31950 tablet_bootstrap.cc:654] T 782f62eb19c14cb1bacdf41fc5855994 P c0551f1684494eb89f0a382480942106: Neither blocks nor log segments found. Creating new log.
I20260811 22:30:13.788643 31950 log.cc:824] T 782f62eb19c14cb1bacdf41fc5855994 P c0551f1684494eb89f0a382480942106: Log is configured to *not* fsync() on all Append() calls
I20260811 22:30:13.789181 31946 tablet_bootstrap.cc:492] T 782f62eb19c14cb1bacdf41fc5855994 P 909609d27239488682586693d2e1cdc1: No bootstrap required, opened a new log
I20260811 22:30:13.789480 31946 ts_tablet_manager.cc:1397] T 782f62eb19c14cb1bacdf41fc5855994 P 909609d27239488682586693d2e1cdc1: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260811 22:30:13.790918 31950 tablet_bootstrap.cc:492] T 782f62eb19c14cb1bacdf41fc5855994 P c0551f1684494eb89f0a382480942106: No bootstrap required, opened a new log
I20260811 22:30:13.791136 31950 ts_tablet_manager.cc:1397] T 782f62eb19c14cb1bacdf41fc5855994 P c0551f1684494eb89f0a382480942106: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260811 22:30:13.795717 31945 raft_consensus.cc:348] T 782f62eb19c14cb1bacdf41fc5855994 P 1096bacf18334b1eba94d3cd4835b68d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0551f1684494eb89f0a382480942106" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42515 } } peers { permanent_uuid: "909609d27239488682586693d2e1cdc1" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39595 } } peers { permanent_uuid: "1096bacf18334b1eba94d3cd4835b68d" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 36713 } }
I20260811 22:30:13.796114 31945 raft_consensus.cc:374] T 782f62eb19c14cb1bacdf41fc5855994 P 1096bacf18334b1eba94d3cd4835b68d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:30:13.796208 31945 raft_consensus.cc:729] T 782f62eb19c14cb1bacdf41fc5855994 P 1096bacf18334b1eba94d3cd4835b68d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1096bacf18334b1eba94d3cd4835b68d, State: Initialized, Role: FOLLOWER
I20260811 22:30:13.796804 31945 consensus_queue.cc:260] T 782f62eb19c14cb1bacdf41fc5855994 P 1096bacf18334b1eba94d3cd4835b68d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0551f1684494eb89f0a382480942106" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42515 } } peers { permanent_uuid: "909609d27239488682586693d2e1cdc1" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39595 } } peers { permanent_uuid: "1096bacf18334b1eba94d3cd4835b68d" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 36713 } }
I20260811 22:30:13.797978 31950 raft_consensus.cc:348] T 782f62eb19c14cb1bacdf41fc5855994 P c0551f1684494eb89f0a382480942106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0551f1684494eb89f0a382480942106" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42515 } } peers { permanent_uuid: "909609d27239488682586693d2e1cdc1" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39595 } } peers { permanent_uuid: "1096bacf18334b1eba94d3cd4835b68d" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 36713 } }
I20260811 22:30:13.798425 31950 raft_consensus.cc:374] T 782f62eb19c14cb1bacdf41fc5855994 P c0551f1684494eb89f0a382480942106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:30:13.798573 31945 ts_tablet_manager.cc:1428] T 782f62eb19c14cb1bacdf41fc5855994 P 1096bacf18334b1eba94d3cd4835b68d: Time spent starting tablet: real 0.013s user 0.012s sys 0.002s
I20260811 22:30:13.798646 31950 raft_consensus.cc:729] T 782f62eb19c14cb1bacdf41fc5855994 P c0551f1684494eb89f0a382480942106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0551f1684494eb89f0a382480942106, State: Initialized, Role: FOLLOWER
I20260811 22:30:13.798887 31435 heartbeater.cc:503] Master 127.29.106.62:42541 was elected leader, sending a full tablet report...
I20260811 22:30:13.799319 31950 consensus_queue.cc:260] T 782f62eb19c14cb1bacdf41fc5855994 P c0551f1684494eb89f0a382480942106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0551f1684494eb89f0a382480942106" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42515 } } peers { permanent_uuid: "909609d27239488682586693d2e1cdc1" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39595 } } peers { permanent_uuid: "1096bacf18334b1eba94d3cd4835b68d" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 36713 } }
I20260811 22:30:13.799372 31946 raft_consensus.cc:348] T 782f62eb19c14cb1bacdf41fc5855994 P 909609d27239488682586693d2e1cdc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0551f1684494eb89f0a382480942106" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42515 } } peers { permanent_uuid: "909609d27239488682586693d2e1cdc1" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39595 } } peers { permanent_uuid: "1096bacf18334b1eba94d3cd4835b68d" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 36713 } }
I20260811 22:30:13.799713 31946 raft_consensus.cc:374] T 782f62eb19c14cb1bacdf41fc5855994 P 909609d27239488682586693d2e1cdc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:30:13.799856 31946 raft_consensus.cc:729] T 782f62eb19c14cb1bacdf41fc5855994 P 909609d27239488682586693d2e1cdc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 909609d27239488682586693d2e1cdc1, State: Initialized, Role: FOLLOWER
I20260811 22:30:13.800343 31912 heartbeater.cc:503] Master 127.29.106.62:42541 was elected leader, sending a full tablet report...
I20260811 22:30:13.800631 31946 consensus_queue.cc:260] T 782f62eb19c14cb1bacdf41fc5855994 P 909609d27239488682586693d2e1cdc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0551f1684494eb89f0a382480942106" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42515 } } peers { permanent_uuid: "909609d27239488682586693d2e1cdc1" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39595 } } peers { permanent_uuid: "1096bacf18334b1eba94d3cd4835b68d" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 36713 } }
I20260811 22:30:13.800915 31950 ts_tablet_manager.cc:1428] T 782f62eb19c14cb1bacdf41fc5855994 P c0551f1684494eb89f0a382480942106: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260811 22:30:13.801818 31663 heartbeater.cc:503] Master 127.29.106.62:42541 was elected leader, sending a full tablet report...
I20260811 22:30:13.803952 31946 ts_tablet_manager.cc:1428] T 782f62eb19c14cb1bacdf41fc5855994 P 909609d27239488682586693d2e1cdc1: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
W20260811 22:30:13.806555 31436 tablet.cc:2369] T 782f62eb19c14cb1bacdf41fc5855994 P 1096bacf18334b1eba94d3cd4835b68d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:30:13.837762 31956 raft_consensus.cc:482] T 782f62eb19c14cb1bacdf41fc5855994 P c0551f1684494eb89f0a382480942106 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:30:13.837996 31956 raft_consensus.cc:504] T 782f62eb19c14cb1bacdf41fc5855994 P c0551f1684494eb89f0a382480942106 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0551f1684494eb89f0a382480942106" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42515 } } peers { permanent_uuid: "909609d27239488682586693d2e1cdc1" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39595 } } peers { permanent_uuid: "1096bacf18334b1eba94d3cd4835b68d" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 36713 } }
I20260811 22:30:13.839582 31956 leader_election.cc:288] T 782f62eb19c14cb1bacdf41fc5855994 P c0551f1684494eb89f0a382480942106 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 909609d27239488682586693d2e1cdc1 (127.29.106.2:39595), 1096bacf18334b1eba94d3cd4835b68d (127.29.106.1:36713)
I20260811 22:30:13.846225 31944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:13.840121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:42413 (local address 127.29.106.1:36713)
0811 22:30:13.840313 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:13.840320 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:13.840341 (+ 21us) server_negotiation.cc:407] Connection header received
0811 22:30:13.840412 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:13.840416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:13.840497 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:13.840608 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:30:13.841105 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:13.842014 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:13.844307 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:13.845367 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:13.845515 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:13.845581 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:13.845874 (+ 293us) server_negotiation.cc:299] Negotiation successful
0811 22:30:13.845993 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260811 22:30:13.846274 31916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:13.839842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:36713 (local address 127.29.106.3:42413)
0811 22:30:13.840098 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:13.840117 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:13.840235 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:13.840657 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:13.840662 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:13.840691 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:30:13.840951 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:13.840961 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:13.842145 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:13.842149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:13.844159 (+ 2010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:13.844179 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:13.845512 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:13.845518 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:13.845680 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:13.845714 (+ 34us) client_negotiation.cc:769] Sending connection context
0811 22:30:13.845803 (+ 89us) client_negotiation.cc:240] Negotiation successful
0811 22:30:13.845955 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"spinlock_wait_cycles":30464}
I20260811 22:30:13.846698 31943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:13.839991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:57497 (local address 127.29.106.2:39595)
0811 22:30:13.840149 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:13.840155 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:13.840751 (+ 596us) server_negotiation.cc:407] Connection header received
0811 22:30:13.840907 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:13.840911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:13.840962 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:13.841033 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:30:13.841509 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:13.842386 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:13.844619 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:13.845723 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:13.845864 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:13.845926 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:13.846396 (+ 470us) server_negotiation.cc:299] Negotiation successful
0811 22:30:13.846512 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260811 22:30:13.847193 31958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:13.839917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.2:39595 (local address 127.29.106.3:57497)
0811 22:30:13.840652 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:13.840669 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:13.840798 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:13.841074 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:13.841077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:13.841096 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:30:13.841360 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:13.841367 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:13.842528 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:13.842533 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:13.844467 (+ 1934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:13.844478 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:13.846045 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:13.846052 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:13.846209 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:13.846232 (+ 23us) client_negotiation.cc:769] Sending connection context
0811 22:30:13.846847 (+ 615us) client_negotiation.cc:240] Negotiation successful
0811 22:30:13.846963 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"mutex_wait_us":2,"thread_start_us":434,"threads_started":1}
I20260811 22:30:13.847172 31351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "782f62eb19c14cb1bacdf41fc5855994" candidate_uuid: "c0551f1684494eb89f0a382480942106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1096bacf18334b1eba94d3cd4835b68d" is_pre_election: true
I20260811 22:30:13.847704 31351 raft_consensus.cc:2454] T 782f62eb19c14cb1bacdf41fc5855994 P 1096bacf18334b1eba94d3cd4835b68d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0551f1684494eb89f0a382480942106 in term 0.
I20260811 22:30:13.848145 31590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "782f62eb19c14cb1bacdf41fc5855994" candidate_uuid: "c0551f1684494eb89f0a382480942106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "909609d27239488682586693d2e1cdc1" is_pre_election: true
I20260811 22:30:13.848318 31699 leader_election.cc:302] T 782f62eb19c14cb1bacdf41fc5855994 P c0551f1684494eb89f0a382480942106 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1096bacf18334b1eba94d3cd4835b68d, c0551f1684494eb89f0a382480942106; no voters:
I20260811 22:30:13.848565 31590 raft_consensus.cc:2454] T 782f62eb19c14cb1bacdf41fc5855994 P 909609d27239488682586693d2e1cdc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0551f1684494eb89f0a382480942106 in term 0.
I20260811 22:30:13.848680 31956 raft_consensus.cc:2781] T 782f62eb19c14cb1bacdf41fc5855994 P c0551f1684494eb89f0a382480942106 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:30:13.848821 31956 raft_consensus.cc:482] T 782f62eb19c14cb1bacdf41fc5855994 P c0551f1684494eb89f0a382480942106 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:30:13.848925 31956 raft_consensus.cc:3037] T 782f62eb19c14cb1bacdf41fc5855994 P c0551f1684494eb89f0a382480942106 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:30:13.851094 31956 raft_consensus.cc:504] T 782f62eb19c14cb1bacdf41fc5855994 P c0551f1684494eb89f0a382480942106 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0551f1684494eb89f0a382480942106" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42515 } } peers { permanent_uuid: "909609d27239488682586693d2e1cdc1" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39595 } } peers { permanent_uuid: "1096bacf18334b1eba94d3cd4835b68d" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 36713 } }
I20260811 22:30:13.851707 31956 leader_election.cc:288] T 782f62eb19c14cb1bacdf41fc5855994 P c0551f1684494eb89f0a382480942106 [CANDIDATE]: Term 1 election: Requested vote from peers 909609d27239488682586693d2e1cdc1 (127.29.106.2:39595), 1096bacf18334b1eba94d3cd4835b68d (127.29.106.1:36713)
I20260811 22:30:13.852052 31590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "782f62eb19c14cb1bacdf41fc5855994" candidate_uuid: "c0551f1684494eb89f0a382480942106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "909609d27239488682586693d2e1cdc1"
I20260811 22:30:13.852196 31351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "782f62eb19c14cb1bacdf41fc5855994" candidate_uuid: "c0551f1684494eb89f0a382480942106" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1096bacf18334b1eba94d3cd4835b68d"
I20260811 22:30:13.852241 31590 raft_consensus.cc:3037] T 782f62eb19c14cb1bacdf41fc5855994 P 909609d27239488682586693d2e1cdc1 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:30:13.852376 31351 raft_consensus.cc:3037] T 782f62eb19c14cb1bacdf41fc5855994 P 1096bacf18334b1eba94d3cd4835b68d [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:30:13.854364 31590 raft_consensus.cc:2454] T 782f62eb19c14cb1bacdf41fc5855994 P 909609d27239488682586693d2e1cdc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0551f1684494eb89f0a382480942106 in term 1.
I20260811 22:30:13.854516 31351 raft_consensus.cc:2454] T 782f62eb19c14cb1bacdf41fc5855994 P 1096bacf18334b1eba94d3cd4835b68d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0551f1684494eb89f0a382480942106 in term 1.
I20260811 22:30:13.854722 31698 leader_election.cc:302] T 782f62eb19c14cb1bacdf41fc5855994 P c0551f1684494eb89f0a382480942106 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 909609d27239488682586693d2e1cdc1, c0551f1684494eb89f0a382480942106; no voters:
I20260811 22:30:13.855017 31956 raft_consensus.cc:2781] T 782f62eb19c14cb1bacdf41fc5855994 P c0551f1684494eb89f0a382480942106 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:30:13.855456 31956 raft_consensus.cc:686] T 782f62eb19c14cb1bacdf41fc5855994 P c0551f1684494eb89f0a382480942106 [term 1 LEADER]: Becoming Leader. State: Replica: c0551f1684494eb89f0a382480942106, State: Running, Role: LEADER
I20260811 22:30:13.855945 31956 consensus_queue.cc:237] T 782f62eb19c14cb1bacdf41fc5855994 P c0551f1684494eb89f0a382480942106 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0551f1684494eb89f0a382480942106" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42515 } } peers { permanent_uuid: "909609d27239488682586693d2e1cdc1" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39595 } } peers { permanent_uuid: "1096bacf18334b1eba94d3cd4835b68d" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 36713 } }
I20260811 22:30:13.860085 31107 catalog_manager.cc:5495] T 782f62eb19c14cb1bacdf41fc5855994 P c0551f1684494eb89f0a382480942106 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0551f1684494eb89f0a382480942106 (127.29.106.3). New cstate: current_term: 1 leader_uuid: "c0551f1684494eb89f0a382480942106" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0551f1684494eb89f0a382480942106" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 42515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "909609d27239488682586693d2e1cdc1" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1096bacf18334b1eba94d3cd4835b68d" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 36713 } health_report { overall_health: UNKNOWN } } }
I20260811 22:30:13.880596 31938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:13.875387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54506 (local address 127.29.106.3:42515)
0811 22:30:13.875616 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:13.875622 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:13.875718 (+ 96us) server_negotiation.cc:407] Connection header received
0811 22:30:13.875922 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:13.875926 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:13.876004 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:13.876124 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:30:13.876484 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:13.877186 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:13.878613 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:13.879004 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:13.879107 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:13.879631 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:30:13.880065 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:30:13.880202 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:13.880292 (+ 90us) server_negotiation.cc:299] Negotiation successful
0811 22:30:13.880366 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260811 22:30:13.887777 31351 raft_consensus.cc:1261] T 782f62eb19c14cb1bacdf41fc5855994 P 1096bacf18334b1eba94d3cd4835b68d [term 1 FOLLOWER]: Refusing update from remote peer c0551f1684494eb89f0a382480942106: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:30:13.887843 31590 raft_consensus.cc:1261] T 782f62eb19c14cb1bacdf41fc5855994 P 909609d27239488682586693d2e1cdc1 [term 1 FOLLOWER]: Refusing update from remote peer c0551f1684494eb89f0a382480942106: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:30:13.888715 31959 consensus_queue.cc:1035] T 782f62eb19c14cb1bacdf41fc5855994 P c0551f1684494eb89f0a382480942106 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1096bacf18334b1eba94d3cd4835b68d" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 36713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:30:13.889065 31956 consensus_queue.cc:1035] T 782f62eb19c14cb1bacdf41fc5855994 P c0551f1684494eb89f0a382480942106 [LEADER]: Connected to new peer: Peer: permanent_uuid: "909609d27239488682586693d2e1cdc1" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 39595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:30:13.902047 31964 mvcc.cc:205] Tried to move back new op lower bound from 7317452447275876352 to 7317452447157714944. Current Snapshot: MvccSnapshot[applied={T|T < 7317452447275876352}]
I20260811 22:30:13.907627 31965 mvcc.cc:205] Tried to move back new op lower bound from 7317452447275876352 to 7317452447157714944. Current Snapshot: MvccSnapshot[applied={T|T < 7317452447275876352}]
I20260811 22:30:13.917578 31106 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:30:13.917930 31106 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:30:13.921402 31106 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d2defc52470e47ce863e6b5d28468b14: Sending DeleteTablet for 3 replicas of tablet 782f62eb19c14cb1bacdf41fc5855994
I20260811 22:30:13.922422 31552 tablet_service.cc:1514] Processing DeleteTablet for tablet 782f62eb19c14cb1bacdf41fc5855994 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:30:13 UTC) from {username='slave'} at 127.0.0.1:36074
I20260811 22:30:13.922858 31793 tablet_service.cc:1514] Processing DeleteTablet for tablet 782f62eb19c14cb1bacdf41fc5855994 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:30:13 UTC) from {username='slave'} at 127.0.0.1:54494
I20260811 22:30:13.923298 31981 tablet_replica.cc:330] T 782f62eb19c14cb1bacdf41fc5855994 P 909609d27239488682586693d2e1cdc1: stopping tablet replica
I20260811 22:30:13.923303 31317 tablet_service.cc:1514] Processing DeleteTablet for tablet 782f62eb19c14cb1bacdf41fc5855994 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:30:13 UTC) from {username='slave'} at 127.0.0.1:56576
I20260811 22:30:13.923785 31981 raft_consensus.cc:2229] T 782f62eb19c14cb1bacdf41fc5855994 P 909609d27239488682586693d2e1cdc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:30:13.924278 31981 raft_consensus.cc:2258] T 782f62eb19c14cb1bacdf41fc5855994 P 909609d27239488682586693d2e1cdc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:30:13.924530 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 31204
I20260811 22:30:13.926365 31981 ts_tablet_manager.cc:1905] T 782f62eb19c14cb1bacdf41fc5855994 P 909609d27239488682586693d2e1cdc1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:30:13.932533 31984 tablet_replica.cc:330] T 782f62eb19c14cb1bacdf41fc5855994 P 1096bacf18334b1eba94d3cd4835b68d: stopping tablet replica
I20260811 22:30:13.932938 31984 raft_consensus.cc:2229] T 782f62eb19c14cb1bacdf41fc5855994 P 1096bacf18334b1eba94d3cd4835b68d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:30:13.933435 31984 raft_consensus.cc:2258] T 782f62eb19c14cb1bacdf41fc5855994 P 1096bacf18334b1eba94d3cd4835b68d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:30:13.935650 31981 ts_tablet_manager.cc:1918] T 782f62eb19c14cb1bacdf41fc5855994 P 909609d27239488682586693d2e1cdc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:30:13.935873 31981 log.cc:1196] T 782f62eb19c14cb1bacdf41fc5855994 P 909609d27239488682586693d2e1cdc1: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-1/wal/wals/782f62eb19c14cb1bacdf41fc5855994
I20260811 22:30:13.936496 31981 ts_tablet_manager.cc:1939] T 782f62eb19c14cb1bacdf41fc5855994 P 909609d27239488682586693d2e1cdc1: Deleting consensus metadata
I20260811 22:30:13.938174 31984 ts_tablet_manager.cc:1905] T 782f62eb19c14cb1bacdf41fc5855994 P 1096bacf18334b1eba94d3cd4835b68d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:30:13.938419 31083 catalog_manager.cc:4841] TS 909609d27239488682586693d2e1cdc1 (127.29.106.2:39595): tablet 782f62eb19c14cb1bacdf41fc5855994 (table test-table [id=774432f14f6543ff8f46b05e8bd56727]) successfully deleted
I20260811 22:30:13.939653 31982 tablet_replica.cc:330] T 782f62eb19c14cb1bacdf41fc5855994 P c0551f1684494eb89f0a382480942106: stopping tablet replica
I20260811 22:30:13.940019 31982 raft_consensus.cc:2229] T 782f62eb19c14cb1bacdf41fc5855994 P c0551f1684494eb89f0a382480942106 [term 1 LEADER]: Raft consensus shutting down.
I20260811 22:30:13.940632 31982 raft_consensus.cc:2258] T 782f62eb19c14cb1bacdf41fc5855994 P c0551f1684494eb89f0a382480942106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:30:13.942603 31982 ts_tablet_manager.cc:1905] T 782f62eb19c14cb1bacdf41fc5855994 P c0551f1684494eb89f0a382480942106: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:30:13.945230 31984 ts_tablet_manager.cc:1918] T 782f62eb19c14cb1bacdf41fc5855994 P 1096bacf18334b1eba94d3cd4835b68d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:30:13.945391 31984 log.cc:1196] T 782f62eb19c14cb1bacdf41fc5855994 P 1096bacf18334b1eba94d3cd4835b68d: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-0/wal/wals/782f62eb19c14cb1bacdf41fc5855994
I20260811 22:30:13.945940 31984 ts_tablet_manager.cc:1939] T 782f62eb19c14cb1bacdf41fc5855994 P 1096bacf18334b1eba94d3cd4835b68d: Deleting consensus metadata
I20260811 22:30:13.947745 31944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:13.926958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56588 (local address 127.29.106.1:36713)
0811 22:30:13.927133 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:13.927139 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:13.938983 (+ 11844us) server_negotiation.cc:407] Connection header received
0811 22:30:13.939177 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:13.939183 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:13.939393 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:13.939493 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:13.940286 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:13.941388 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:13.943401 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:13.944060 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:13.944176 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:13.945721 (+ 1545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:13.945732 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:13.945749 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:30:13.946589 (+ 840us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:13.946665 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:30:13.946690 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:13.946790 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:13.947259 (+ 469us) server_negotiation.cc:299] Negotiation successful
0811 22:30:13.947387 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260811 22:30:13.947810 31084 catalog_manager.cc:4841] TS 1096bacf18334b1eba94d3cd4835b68d (127.29.106.1:36713): tablet 782f62eb19c14cb1bacdf41fc5855994 (table test-table [id=774432f14f6543ff8f46b05e8bd56727]) successfully deleted
I20260811 22:30:13.948551 31427 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:30:13.949743 31982 ts_tablet_manager.cc:1918] T 782f62eb19c14cb1bacdf41fc5855994 P c0551f1684494eb89f0a382480942106: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:30:13.949976 31982 log.cc:1196] T 782f62eb19c14cb1bacdf41fc5855994 P c0551f1684494eb89f0a382480942106: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786487105757681-30120-0/minicluster-data/ts-2/wal/wals/782f62eb19c14cb1bacdf41fc5855994
I20260811 22:30:13.950704 31982 ts_tablet_manager.cc:1939] T 782f62eb19c14cb1bacdf41fc5855994 P c0551f1684494eb89f0a382480942106: Deleting consensus metadata
I20260811 22:30:13.952529 31081 catalog_manager.cc:4841] TS c0551f1684494eb89f0a382480942106 (127.29.106.3:42515): tablet 782f62eb19c14cb1bacdf41fc5855994 (table test-table [id=774432f14f6543ff8f46b05e8bd56727]) successfully deleted
I20260811 22:30:14.036010 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 31204
I20260811 22:30:14.061249 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 31439
I20260811 22:30:14.068629 31943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:14.062096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36080 (local address 127.29.106.2:39595)
0811 22:30:14.062278 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:14.062283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:14.062301 (+ 18us) server_negotiation.cc:407] Connection header received
0811 22:30:14.062465 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:14.062470 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:14.062580 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:14.062662 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:14.063110 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:14.063847 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:14.065720 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:14.066103 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:14.066191 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:14.067257 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:14.067361 (+ 104us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:14.067381 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:30:14.067964 (+ 583us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:14.068013 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:30:14.068033 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:14.068124 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:14.068255 (+ 131us) server_negotiation.cc:299] Negotiation successful
0811 22:30:14.068338 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260811 22:30:14.069295 31655 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:30:14.158216 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 31439
I20260811 22:30:14.193594 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 31667
I20260811 22:30:14.200660 31938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:14.194363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54508 (local address 127.29.106.3:42515)
0811 22:30:14.194539 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:14.194545 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:14.194635 (+ 90us) server_negotiation.cc:407] Connection header received
0811 22:30:14.194839 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:14.194843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:14.194942 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:14.195029 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:14.195516 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:14.196358 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:14.197635 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:14.198084 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:14.198193 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:14.199226 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:14.199235 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:14.199254 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:30:14.199903 (+ 649us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:14.199951 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:30:14.199973 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:14.200078 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:14.200193 (+ 115us) server_negotiation.cc:299] Negotiation successful
0811 22:30:14.200294 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260811 22:30:14.201359 31902 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:30:14.320236 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 31667
I20260811 22:30:14.349326 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 31050
I20260811 22:30:14.350327 31163 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:30:14.458521 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 31050
2026-08-11T22:30:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2700 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:30:14 dist-test-slave-gjrz krb5kdc[32005](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:30:14 dist-test-slave-gjrz krb5kdc[32005](info): set up 2 sockets
Aug 11 22:30:14 dist-test-slave-gjrz krb5kdc[32005](info): commencing operation
krb5kdc: starting...
W20260811 22:30:15.745218 30120 mini_kdc.cc:121] Time spent starting KDC: real 1.238s user 0.006s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Aug 11 22:30:15 dist-test-slave-gjrz krb5kdc[32005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:30:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:30:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.106.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
I20260811 22:30:16.010746 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:38057
--webserver_interface=127.29.106.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:40809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:38057
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:30:16.340739 32025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:30:16.341013 32025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:30:16.341126 32025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:30:16.348098 32025 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:30:16.348198 32025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:30:16.348253 32025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:30:16.348307 32025 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:30:16.348349 32025 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:30:16.358633 32025 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:40809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:38057
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:38057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.106.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:30:16.359813 32025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:30:16.361021 32025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:30:16.371995 32030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:16.372025 32033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:16.372340 32037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:30:16.373065 32025 server_base.cc:1034] running on GCE node
I20260811 22:30:16.373759 32025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:30:16.375044 32025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:30:16.376246 32025 hybrid_clock.cc:648] HybridClock initialized: now 1786487416376173 us; error 57 us; skew 500 ppm
I20260811 22:30:16.376570 32025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:30:16 dist-test-slave-gjrz krb5kdc[32005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487416, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:30:16.379492 32025 init.cc:377] Logged in from keytab as kudu/127.29.106.62@KRBTEST.COM (short username kudu)
I20260811 22:30:16.380805 32025 webserver.cc:460] Webserver started at http://127.29.106.62:45541/ using document root <none> and password file <none>
I20260811 22:30:16.381395 32025 fs_manager.cc:359] Metadata directory not provided
I20260811 22:30:16.381515 32025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:30:16.381758 32025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:30:16.384609 32025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "cd71674c1d2d43609b8376bd0862f2f5"
format_stamp: "Formatted at 2026-08-11 22:30:16 on dist-test-slave-gjrz"
server_key: "d9f7a2ec4b944cad182460f64856ead6"
server_key_iv: "e019e7451794b6f639e7a23f9582ddee"
server_key_version: "encryptionkey@0"
I20260811 22:30:16.385314 32025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "cd71674c1d2d43609b8376bd0862f2f5"
format_stamp: "Formatted at 2026-08-11 22:30:16 on dist-test-slave-gjrz"
server_key: "d9f7a2ec4b944cad182460f64856ead6"
server_key_iv: "e019e7451794b6f639e7a23f9582ddee"
server_key_version: "encryptionkey@0"
I20260811 22:30:16.390086 32025 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 22:30:16.393234 32052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:16.394404 32025 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 22:30:16.394548 32025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "cd71674c1d2d43609b8376bd0862f2f5"
format_stamp: "Formatted at 2026-08-11 22:30:16 on dist-test-slave-gjrz"
server_key: "d9f7a2ec4b944cad182460f64856ead6"
server_key_iv: "e019e7451794b6f639e7a23f9582ddee"
server_key_version: "encryptionkey@0"
I20260811 22:30:16.394713 32025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:30:16.421406 32025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:30:16.425071 32025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:30:16.425308 32025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:30:16.442852 32025 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.62:38057
I20260811 22:30:16.442888 32145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.62:38057 every 8 connection(s)
I20260811 22:30:16.444160 32025 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:30:16.447022 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 32025
I20260811 22:30:16.447171 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:30:16.447631 30120 external_mini_cluster.cc:1428] Setting key f3dd88c661be6687320e4adc627cc0fc
I20260811 22:30:16.447914 32146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:30:16.461628 32146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd71674c1d2d43609b8376bd0862f2f5: Bootstrap starting.
Aug 11 22:30:16 dist-test-slave-gjrz krb5kdc[32005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:30:16.464802 32146 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd71674c1d2d43609b8376bd0862f2f5: Neither blocks nor log segments found. Creating new log.
I20260811 22:30:16.465819 32146 log.cc:824] T 00000000000000000000000000000000 P cd71674c1d2d43609b8376bd0862f2f5: Log is configured to *not* fsync() on all Append() calls
I20260811 22:30:16.468554 32146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd71674c1d2d43609b8376bd0862f2f5: No bootstrap required, opened a new log
I20260811 22:30:16.470536 32153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:16.449355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38940 (local address 127.29.106.62:38057)
0811 22:30:16.450097 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:16.450107 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:16.450145 (+ 38us) server_negotiation.cc:407] Connection header received
0811 22:30:16.452309 (+ 2164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:16.452328 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:16.452719 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:16.453024 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:16.454954 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:16.456302 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:16.458202 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:16.458891 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:16.459027 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:16.463719 (+ 4692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:16.463726 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:16.463740 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:30:16.463778 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:16.466735 (+ 2957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:16.467416 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:16.467422 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:16.467428 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:16.467591 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:16.467967 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:16.467971 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:16.467980 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:16.468454 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:16.468749 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:16.469205 (+ 456us) server_negotiation.cc:299] Negotiation successful
0811 22:30:16.469608 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":331,"threads_started":1}
I20260811 22:30:16.476110 32146 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd71674c1d2d43609b8376bd0862f2f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd71674c1d2d43609b8376bd0862f2f5" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 38057 } }
I20260811 22:30:16.476436 32146 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd71674c1d2d43609b8376bd0862f2f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:30:16.476526 32146 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd71674c1d2d43609b8376bd0862f2f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd71674c1d2d43609b8376bd0862f2f5, State: Initialized, Role: FOLLOWER
I20260811 22:30:16.476999 32146 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd71674c1d2d43609b8376bd0862f2f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd71674c1d2d43609b8376bd0862f2f5" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 38057 } }
I20260811 22:30:16.477216 32146 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd71674c1d2d43609b8376bd0862f2f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:30:16.477336 32146 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd71674c1d2d43609b8376bd0862f2f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:30:16.477484 32146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd71674c1d2d43609b8376bd0862f2f5 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:30:16.479543 32146 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd71674c1d2d43609b8376bd0862f2f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd71674c1d2d43609b8376bd0862f2f5" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 38057 } }
I20260811 22:30:16.479984 32146 leader_election.cc:302] T 00000000000000000000000000000000 P cd71674c1d2d43609b8376bd0862f2f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd71674c1d2d43609b8376bd0862f2f5; no voters:
I20260811 22:30:16.480525 32146 leader_election.cc:288] T 00000000000000000000000000000000 P cd71674c1d2d43609b8376bd0862f2f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:30:16.480645 32155 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd71674c1d2d43609b8376bd0862f2f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:30:16.481276 32155 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd71674c1d2d43609b8376bd0862f2f5 [term 1 LEADER]: Becoming Leader. State: Replica: cd71674c1d2d43609b8376bd0862f2f5, State: Running, Role: LEADER
I20260811 22:30:16.481781 32155 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd71674c1d2d43609b8376bd0862f2f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd71674c1d2d43609b8376bd0862f2f5" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 38057 } }
I20260811 22:30:16.482420 32146 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd71674c1d2d43609b8376bd0862f2f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:30:16.484972 32158 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd71674c1d2d43609b8376bd0862f2f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd71674c1d2d43609b8376bd0862f2f5. Latest consensus state: current_term: 1 leader_uuid: "cd71674c1d2d43609b8376bd0862f2f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd71674c1d2d43609b8376bd0862f2f5" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 38057 } } }
I20260811 22:30:16.485366 32158 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd71674c1d2d43609b8376bd0862f2f5 [sys.catalog]: This master's current role is: LEADER
I20260811 22:30:16.485980 32157 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd71674c1d2d43609b8376bd0862f2f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd71674c1d2d43609b8376bd0862f2f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd71674c1d2d43609b8376bd0862f2f5" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 38057 } } }
I20260811 22:30:16.486296 32157 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd71674c1d2d43609b8376bd0862f2f5 [sys.catalog]: This master's current role is: LEADER
I20260811 22:30:16.490658 32168 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:30:16.496156 32168 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:30:16.506260 32168 catalog_manager.cc:1347] Generated new cluster ID: a274d68a1d854f699d5e4ddb79baa0a8
I20260811 22:30:16.506372 32168 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:30:16.540433 32168 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:30:16.542346 32168 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:30:16.549368 32168 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd71674c1d2d43609b8376bd0862f2f5: Generated new TSK 0
I20260811 22:30:16.550304 32168 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.106.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab.
Entry for principal kudu/127.29.106.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab.
I20260811 22:30:16.616199 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.1:0
--local_ip_for_outbound_sockets=127.29.106.1
--webserver_interface=127.29.106.1
--webserver_port=0
--tserver_master_addrs=127.29.106.62:38057
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab
--principal=kudu/127.29.106.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:40809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:30:16.960043 32199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:30:16.960299 32199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:30:16.960397 32199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:30:16.967820 32199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:30:16.967916 32199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:30:16.968041 32199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.1
I20260811 22:30:16.977238 32199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:40809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab
--principal=kudu/127.29.106.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.106.1
--webserver_port=0
--tserver_master_addrs=127.29.106.62:38057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.1
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:30:16.978346 32199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:30:16.979575 32199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:30:16.991952 32210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:16.991997 32208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:16.992163 32207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:30:16.993026 32199 server_base.cc:1034] running on GCE node
I20260811 22:30:16.993513 32199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:30:16.994313 32199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:30:16.995519 32199 hybrid_clock.cc:648] HybridClock initialized: now 1786487416995459 us; error 42 us; skew 500 ppm
I20260811 22:30:16.995882 32199 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:30:16 dist-test-slave-gjrz krb5kdc[32005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487416, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:30:16.998853 32199 init.cc:377] Logged in from keytab as kudu/127.29.106.1@KRBTEST.COM (short username kudu)
I20260811 22:30:17.000228 32199 webserver.cc:460] Webserver started at http://127.29.106.1:37353/ using document root <none> and password file <none>
I20260811 22:30:17.000829 32199 fs_manager.cc:359] Metadata directory not provided
I20260811 22:30:17.000953 32199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:30:17.001253 32199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:30:17.004048 32199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-0/data/instance:
uuid: "34aa3b0820ae493a8236de282519a94b"
format_stamp: "Formatted at 2026-08-11 22:30:16 on dist-test-slave-gjrz"
server_key: "a0171fa2f6b8c18dce84fb8a85336422"
server_key_iv: "e7f01fdbd250377ac62c348936709e58"
server_key_version: "encryptionkey@0"
I20260811 22:30:17.004796 32199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance:
uuid: "34aa3b0820ae493a8236de282519a94b"
format_stamp: "Formatted at 2026-08-11 22:30:16 on dist-test-slave-gjrz"
server_key: "a0171fa2f6b8c18dce84fb8a85336422"
server_key_iv: "e7f01fdbd250377ac62c348936709e58"
server_key_version: "encryptionkey@0"
I20260811 22:30:17.009704 32199 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260811 22:30:17.012977 32223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:17.014158 32199 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 22:30:17.014302 32199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-0/wal
uuid: "34aa3b0820ae493a8236de282519a94b"
format_stamp: "Formatted at 2026-08-11 22:30:16 on dist-test-slave-gjrz"
server_key: "a0171fa2f6b8c18dce84fb8a85336422"
server_key_iv: "e7f01fdbd250377ac62c348936709e58"
server_key_version: "encryptionkey@0"
I20260811 22:30:17.014488 32199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:30:17.030455 32199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:30:17.034266 32199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:30:17.034509 32199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:30:17.035382 32199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:30:17.036607 32199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:30:17.036682 32199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:17.036798 32199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:30:17.036851 32199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:17.066265 32199 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.1:44745
I20260811 22:30:17.066300 32420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.1:44745 every 8 connection(s)
I20260811 22:30:17.067473 32199 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
I20260811 22:30:17.070569 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 32199
I20260811 22:30:17.070757 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance
I20260811 22:30:17.071350 30120 external_mini_cluster.cc:1428] Setting key 8a3d3588dc92eba7e4aed1a0af194e08
Aug 11 22:30:17 dist-test-slave-gjrz krb5kdc[32005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487416, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.1@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:30:17.087041 32424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:17.069778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:36797 (local address 127.29.106.62:38057)
0811 22:30:17.070202 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:17.070206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:17.071193 (+ 987us) server_negotiation.cc:407] Connection header received
0811 22:30:17.073394 (+ 2201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:17.073400 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:17.073491 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:17.073612 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:17.074959 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:17.075998 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:17.077468 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:17.077976 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:17.078095 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:17.081425 (+ 3330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:17.081432 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:17.081436 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:30:17.081483 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:17.084411 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:17.085054 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:17.085058 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:17.085061 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:17.085210 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:17.085575 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:17.085580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:17.085583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:17.085892 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:17.086029 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:17.086609 (+ 580us) server_negotiation.cc:299] Negotiation successful
0811 22:30:17.086775 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":202,"threads_started":1}
I20260811 22:30:17.088222 32426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:17.070262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:38057 (local address 127.29.106.1:36797)
0811 22:30:17.071031 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:17.071082 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:17.073093 (+ 2011us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:17.073789 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:17.073797 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:17.074267 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:30:17.074792 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:17.074808 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:17.076138 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:17.076142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:17.077326 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:17.077337 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:17.078179 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:17.078184 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:17.078384 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:17.078977 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:30:17.078999 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:30:17.081217 (+ 2218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:30:17.084550 (+ 3333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:30:17.084556 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:30:17.084567 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:30:17.084949 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:30:17.085308 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:30:17.085311 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:30:17.085313 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:30:17.085481 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:30:17.085992 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:30:17.085997 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:30:17.086276 (+ 279us) client_negotiation.cc:769] Sending connection context
0811 22:30:17.086484 (+ 208us) client_negotiation.cc:240] Negotiation successful
0811 22:30:17.086955 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":311,"threads_started":1}
I20260811 22:30:17.089738 32421 heartbeater.cc:347] Connected to a master server at 127.29.106.62:38057
I20260811 22:30:17.090019 32421 heartbeater.cc:464] Registering TS with master...
I20260811 22:30:17.090691 32421 heartbeater.cc:511] Master 127.29.106.62:38057 requested a full tablet report, sending...
I20260811 22:30:17.092728 32085 ts_manager.cc:194] Registered new tserver with Master: 34aa3b0820ae493a8236de282519a94b (127.29.106.1:44745)
I20260811 22:30:17.094142 32085 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.1@KRBTEST.COM'} at 127.29.106.1:36797
WARNING: no policy specified for kudu/127.29.106.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab.
Entry for principal kudu/127.29.106.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab.
I20260811 22:30:17.133834 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.2:0
--local_ip_for_outbound_sockets=127.29.106.2
--webserver_interface=127.29.106.2
--webserver_port=0
--tserver_master_addrs=127.29.106.62:38057
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab
--principal=kudu/127.29.106.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:40809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:30:17.453190 32431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:30:17.453441 32431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:30:17.453581 32431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:30:17.460556 32431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:30:17.460667 32431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:30:17.460813 32431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.2
I20260811 22:30:17.469978 32431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:40809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab
--principal=kudu/127.29.106.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.106.2
--webserver_port=0
--tserver_master_addrs=127.29.106.62:38057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.2
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:30:17.471122 32431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:30:17.472409 32431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:30:17.484751 32437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:17.484797 32441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:17.484830 32438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:30:17.485528 32431 server_base.cc:1034] running on GCE node
I20260811 22:30:17.486104 32431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:30:17.486922 32431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:30:17.488119 32431 hybrid_clock.cc:648] HybridClock initialized: now 1786487417488072 us; error 44 us; skew 500 ppm
I20260811 22:30:17.488402 32431 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:30:17 dist-test-slave-gjrz krb5kdc[32005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487417, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:30:17.498713 32431 init.cc:377] Logged in from keytab as kudu/127.29.106.2@KRBTEST.COM (short username kudu)
I20260811 22:30:17.500188 32431 webserver.cc:460] Webserver started at http://127.29.106.2:36825/ using document root <none> and password file <none>
I20260811 22:30:17.500811 32431 fs_manager.cc:359] Metadata directory not provided
I20260811 22:30:17.500927 32431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:30:17.501224 32431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:30:17.504046 32431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-1/data/instance:
uuid: "af25b377ab324b67b36aec602ec84afd"
format_stamp: "Formatted at 2026-08-11 22:30:17 on dist-test-slave-gjrz"
server_key: "a4eb6e1c55ca472cb3f4dac3d34b4b4c"
server_key_iv: "34382568bdba31dab2d2780c268b8688"
server_key_version: "encryptionkey@0"
I20260811 22:30:17.504753 32431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance:
uuid: "af25b377ab324b67b36aec602ec84afd"
format_stamp: "Formatted at 2026-08-11 22:30:17 on dist-test-slave-gjrz"
server_key: "a4eb6e1c55ca472cb3f4dac3d34b4b4c"
server_key_iv: "34382568bdba31dab2d2780c268b8688"
server_key_version: "encryptionkey@0"
I20260811 22:30:17.509593 32431 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 22:30:17.512851 32457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:17.514026 32431 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 22:30:17.514173 32431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-1/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-1/wal
uuid: "af25b377ab324b67b36aec602ec84afd"
format_stamp: "Formatted at 2026-08-11 22:30:17 on dist-test-slave-gjrz"
server_key: "a4eb6e1c55ca472cb3f4dac3d34b4b4c"
server_key_iv: "34382568bdba31dab2d2780c268b8688"
server_key_version: "encryptionkey@0"
I20260811 22:30:17.514336 32431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:30:17.542984 32431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:30:17.546900 32431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:30:17.547140 32431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:30:17.547993 32431 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:30:17.549216 32431 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:30:17.549288 32431 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:17.549366 32431 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:30:17.549443 32431 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:17.579475 32431 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.2:36079
I20260811 22:30:17.579538 32668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.2:36079 every 8 connection(s)
I20260811 22:30:17.580641 32431 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
I20260811 22:30:17.589006 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 32431
I20260811 22:30:17.589185 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance
I20260811 22:30:17.589596 30120 external_mini_cluster.cc:1428] Setting key 8ec144367fe06d0699def0e9f9616166
Aug 11 22:30:17 dist-test-slave-gjrz krb5kdc[32005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487417, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.2@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:30:17.599709 32424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:17.583004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:37337 (local address 127.29.106.62:38057)
0811 22:30:17.583157 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:17.583161 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:17.584408 (+ 1247us) server_negotiation.cc:407] Connection header received
0811 22:30:17.586180 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:17.586186 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:17.586266 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:17.586354 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:17.587839 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:17.588562 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:17.590183 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:17.590765 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:17.590875 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:17.594433 (+ 3558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:17.594438 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:17.594442 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:30:17.594474 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:17.596832 (+ 2358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:17.597492 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:17.597496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:17.597499 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:17.597622 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:17.598021 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:17.598025 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:17.598027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:17.598353 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:17.598483 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:17.599117 (+ 634us) server_negotiation.cc:299] Negotiation successful
0811 22:30:17.599378 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260811 22:30:17.600566 32673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:17.583478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:38057 (local address 127.29.106.2:37337)
0811 22:30:17.584243 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:17.584281 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:17.585905 (+ 1624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:17.586590 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:17.586598 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:17.587092 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:30:17.587671 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:17.587689 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:17.588710 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:17.588714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:17.590042 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:17.590055 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:17.590988 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:17.590997 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:17.591318 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:17.591940 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:30:17.591991 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:30:17.594227 (+ 2236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:30:17.596970 (+ 2743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:30:17.596979 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:30:17.596994 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:30:17.597371 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:30:17.597727 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:30:17.597731 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:30:17.597735 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:30:17.597908 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:30:17.598479 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:30:17.598487 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:30:17.598771 (+ 284us) client_negotiation.cc:769] Sending connection context
0811 22:30:17.598983 (+ 212us) client_negotiation.cc:240] Negotiation successful
0811 22:30:17.599533 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":293,"threads_started":1}
I20260811 22:30:17.601889 32671 heartbeater.cc:347] Connected to a master server at 127.29.106.62:38057
I20260811 22:30:17.602152 32671 heartbeater.cc:464] Registering TS with master...
I20260811 22:30:17.602789 32671 heartbeater.cc:511] Master 127.29.106.62:38057 requested a full tablet report, sending...
I20260811 22:30:17.604215 32085 ts_manager.cc:194] Registered new tserver with Master: af25b377ab324b67b36aec602ec84afd (127.29.106.2:36079)
I20260811 22:30:17.605062 32085 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.2@KRBTEST.COM'} at 127.29.106.2:37337
WARNING: no policy specified for kudu/127.29.106.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab.
Entry for principal kudu/127.29.106.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab.
I20260811 22:30:17.650422 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.3:0
--local_ip_for_outbound_sockets=127.29.106.3
--webserver_interface=127.29.106.3
--webserver_port=0
--tserver_master_addrs=127.29.106.62:38057
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab
--principal=kudu/127.29.106.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:40809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:30:17.969707 32680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:30:17.969978 32680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:30:17.970114 32680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:30:17.977196 32680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:30:17.977319 32680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:30:17.977468 32680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.3
I20260811 22:30:17.986790 32680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:40809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab
--principal=kudu/127.29.106.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.106.3
--webserver_port=0
--tserver_master_addrs=127.29.106.62:38057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.3
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:30:17.987988 32680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:30:17.989221 32680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:30:18.001865 32690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:18.001808 32693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:18.002261 32687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:30:18.002627 32680 server_base.cc:1034] running on GCE node
I20260811 22:30:18.003191 32680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:30:18.004061 32680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:30:18.005260 32680 hybrid_clock.cc:648] HybridClock initialized: now 1786487418005213 us; error 36 us; skew 500 ppm
I20260811 22:30:18.005576 32680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:30:18 dist-test-slave-gjrz krb5kdc[32005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487418, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:30:18.008610 32680 init.cc:377] Logged in from keytab as kudu/127.29.106.3@KRBTEST.COM (short username kudu)
I20260811 22:30:18.009933 32680 webserver.cc:460] Webserver started at http://127.29.106.3:44913/ using document root <none> and password file <none>
I20260811 22:30:18.010517 32680 fs_manager.cc:359] Metadata directory not provided
I20260811 22:30:18.010641 32680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:30:18.010895 32680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:30:18.013643 32680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-2/data/instance:
uuid: "78a614bde28849a7b2c0d19b37ad3a12"
format_stamp: "Formatted at 2026-08-11 22:30:18 on dist-test-slave-gjrz"
server_key: "307003d2cf4d8ac42bdad6f16a45f360"
server_key_iv: "d9705adf08a67c9072909790c70fec30"
server_key_version: "encryptionkey@0"
I20260811 22:30:18.014308 32680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance:
uuid: "78a614bde28849a7b2c0d19b37ad3a12"
format_stamp: "Formatted at 2026-08-11 22:30:18 on dist-test-slave-gjrz"
server_key: "307003d2cf4d8ac42bdad6f16a45f360"
server_key_iv: "d9705adf08a67c9072909790c70fec30"
server_key_version: "encryptionkey@0"
I20260811 22:30:18.019258 32680 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260811 22:30:18.022430 32703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:18.023664 32680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:30:18.023804 32680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-2/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-2/wal
uuid: "78a614bde28849a7b2c0d19b37ad3a12"
format_stamp: "Formatted at 2026-08-11 22:30:18 on dist-test-slave-gjrz"
server_key: "307003d2cf4d8ac42bdad6f16a45f360"
server_key_iv: "d9705adf08a67c9072909790c70fec30"
server_key_version: "encryptionkey@0"
I20260811 22:30:18.023974 32680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:30:18.039670 32680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:30:18.043202 32680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:30:18.043485 32680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:30:18.044384 32680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:30:18.045603 32680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:30:18.045679 32680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:18.045799 32680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:30:18.045859 32680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:18.075631 32680 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.3:46485
I20260811 22:30:18.075681 440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.3:46485 every 8 connection(s)
I20260811 22:30:18.076797 32680 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
I20260811 22:30:18.083657 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 32680
I20260811 22:30:18.083844 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance
I20260811 22:30:18.084371 30120 external_mini_cluster.cc:1428] Setting key 1a5a29f8e567a0ee01f0fcdb406fd94a
Aug 11 22:30:18 dist-test-slave-gjrz krb5kdc[32005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487418, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.3@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:30:18.095739 32424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:18.079055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:59379 (local address 127.29.106.62:38057)
0811 22:30:18.079307 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:18.079313 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:18.080403 (+ 1090us) server_negotiation.cc:407] Connection header received
0811 22:30:18.082275 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:18.082279 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:18.082349 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:18.082438 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:18.084199 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:18.085201 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:18.087068 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:18.087589 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:18.087688 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:18.090825 (+ 3137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:18.090830 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:18.090834 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:30:18.090868 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:18.093334 (+ 2466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:18.093945 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:18.093948 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:18.093951 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:18.094086 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:18.094403 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:18.094406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:18.094408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:18.094654 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:18.094761 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:18.095320 (+ 559us) server_negotiation.cc:299] Negotiation successful
0811 22:30:18.095484 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260811 22:30:18.096565 447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:18.079533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:38057 (local address 127.29.106.3:59379)
0811 22:30:18.080226 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:18.080270 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:18.081946 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:18.082656 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:18.082663 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:18.083211 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:30:18.083980 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:18.084008 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:18.085381 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:18.085387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:18.086918 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:18.086933 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:18.087795 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:18.087800 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:18.088000 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:18.088597 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:30:18.088625 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:30:18.090639 (+ 2014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:30:18.093447 (+ 2808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:30:18.093453 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:30:18.093463 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:30:18.093860 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:30:18.094163 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:30:18.094166 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:30:18.094168 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:30:18.094329 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:30:18.094740 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:30:18.094745 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:30:18.095013 (+ 268us) client_negotiation.cc:769] Sending connection context
0811 22:30:18.095222 (+ 209us) client_negotiation.cc:240] Negotiation successful
0811 22:30:18.095624 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":265,"threads_started":1}
I20260811 22:30:18.096902 32421 heartbeater.cc:503] Master 127.29.106.62:38057 was elected leader, sending a full tablet report...
I20260811 22:30:18.097891 443 heartbeater.cc:347] Connected to a master server at 127.29.106.62:38057
I20260811 22:30:18.098166 443 heartbeater.cc:464] Registering TS with master...
I20260811 22:30:18.098937 443 heartbeater.cc:511] Master 127.29.106.62:38057 requested a full tablet report, sending...
I20260811 22:30:18.100231 32084 ts_manager.cc:194] Registered new tserver with Master: 78a614bde28849a7b2c0d19b37ad3a12 (127.29.106.3:46485)
I20260811 22:30:18.101015 32084 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.3@KRBTEST.COM'} at 127.29.106.3:59379
I20260811 22:30:18.112030 30120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 22:30:18.128687 32424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:18.115925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38946 (local address 127.29.106.62:38057)
0811 22:30:18.116117 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:18.116123 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:18.116483 (+ 360us) server_negotiation.cc:407] Connection header received
0811 22:30:18.116649 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:18.116654 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:18.116713 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:18.116791 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:18.118422 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:18.119212 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:18.120603 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:18.121029 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:18.121138 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:18.123794 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:18.123799 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:18.123802 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:30:18.123831 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:18.125964 (+ 2133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:18.126711 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:18.126715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:18.126717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:18.126855 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:18.127302 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:18.127307 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:18.127311 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:18.127613 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:18.127850 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:18.128242 (+ 392us) server_negotiation.cc:299] Negotiation successful
0811 22:30:18.128411 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 22:30:18.132172 32084 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38946:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:30:18.138758 32084 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:30:18.160388 464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:18.151201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.2:36079 (local address 127.0.0.1:56650)
0811 22:30:18.152810 (+ 1609us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:18.152831 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:18.152923 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:18.153936 (+ 1013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:18.153940 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:18.153965 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:30:18.154370 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:18.154385 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:18.156181 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:18.156185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:18.158104 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:18.158117 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:18.159408 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:18.159415 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:18.159608 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:18.159647 (+ 39us) client_negotiation.cc:769] Sending connection context
0811 22:30:18.159766 (+ 119us) client_negotiation.cc:240] Negotiation successful
0811 22:30:18.159951 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1488,"mutex_wait_us":24,"thread_start_us":1341,"threads_started":1}
I20260811 22:30:18.160692 458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:18.150034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.3:46485 (local address 127.0.0.1:41020)
0811 22:30:18.150825 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:18.150884 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:18.151087 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:18.152897 (+ 1810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:18.152904 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:18.152930 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:30:18.153393 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:18.153407 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:18.155610 (+ 2203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:18.155615 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:18.157965 (+ 2350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:18.157980 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:18.159823 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:18.159829 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:18.159981 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:18.160022 (+ 41us) client_negotiation.cc:769] Sending connection context
0811 22:30:18.160129 (+ 107us) client_negotiation.cc:240] Negotiation successful
0811 22:30:18.160255 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"mutex_wait_us":65,"thread_start_us":421,"threads_started":1}
I20260811 22:30:18.160979 459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:18.150133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:44745 (local address 127.0.0.1:41770)
0811 22:30:18.152519 (+ 2386us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:18.152543 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:18.152669 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:18.153363 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:18.153370 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:18.153393 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:30:18.153728 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:18.153741 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:18.155422 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:18.155427 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:18.158068 (+ 2641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:18.158083 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:18.160132 (+ 2049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:18.160138 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:18.160323 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:18.160356 (+ 33us) client_negotiation.cc:769] Sending connection context
0811 22:30:18.160452 (+ 96us) client_negotiation.cc:240] Negotiation successful
0811 22:30:18.160582 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2232,"mutex_wait_us":58,"thread_start_us":2093,"threads_started":1}
I20260811 22:30:18.161079 466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:18.151705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56650 (local address 127.29.106.2:36079)
0811 22:30:18.153382 (+ 1677us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:18.153389 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:18.153414 (+ 25us) server_negotiation.cc:407] Connection header received
0811 22:30:18.153517 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:18.153524 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:18.153728 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:18.154566 (+ 838us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:30:18.154791 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:18.156046 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:18.158276 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:18.159198 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:18.159349 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:18.159548 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:18.160630 (+ 1082us) server_negotiation.cc:299] Negotiation successful
0811 22:30:18.160808 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1520,"thread_start_us":1418,"threads_started":1}
I20260811 22:30:18.161362 463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:18.150797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41770 (local address 127.29.106.1:44745)
0811 22:30:18.151715 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:18.151724 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:18.152808 (+ 1084us) server_negotiation.cc:407] Connection header received
0811 22:30:18.152920 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:18.152929 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:18.153168 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:18.153344 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:30:18.153931 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:18.155294 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:18.158229 (+ 2935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:18.159681 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:18.159797 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:18.159985 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:18.160904 (+ 919us) server_negotiation.cc:299] Negotiation successful
0811 22:30:18.161078 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":748,"thread_start_us":606,"threads_started":1}
I20260811 22:30:18.161504 462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:18.150286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41020 (local address 127.29.106.3:46485)
0811 22:30:18.152246 (+ 1960us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:18.152253 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:18.152284 (+ 31us) server_negotiation.cc:407] Connection header received
0811 22:30:18.152380 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:18.152387 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:18.152644 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:18.153600 (+ 956us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:30:18.153868 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:18.155474 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:18.158174 (+ 2700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:18.159603 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:18.159732 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:18.159952 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:18.160175 (+ 223us) server_negotiation.cc:299] Negotiation successful
0811 22:30:18.160340 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1784,"thread_start_us":1602,"threads_started":1}
I20260811 22:30:18.164433 32556 tablet_service.cc:1467] Processing CreateTablet for tablet b0999fa2827c49be81f591af0a501081 (DEFAULT_TABLE table=test-table [id=3b8fb122709b41a8a527e7079a304e85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:30:18.165002 332 tablet_service.cc:1467] Processing CreateTablet for tablet b0999fa2827c49be81f591af0a501081 (DEFAULT_TABLE table=test-table [id=3b8fb122709b41a8a527e7079a304e85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:30:18.165562 32556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0999fa2827c49be81f591af0a501081. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:30:18.165783 32310 tablet_service.cc:1467] Processing CreateTablet for tablet b0999fa2827c49be81f591af0a501081 (DEFAULT_TABLE table=test-table [id=3b8fb122709b41a8a527e7079a304e85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:30:18.166414 332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0999fa2827c49be81f591af0a501081. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:30:18.167148 32310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0999fa2827c49be81f591af0a501081. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:30:18.176481 467 tablet_bootstrap.cc:492] T b0999fa2827c49be81f591af0a501081 P af25b377ab324b67b36aec602ec84afd: Bootstrap starting.
I20260811 22:30:18.179440 467 tablet_bootstrap.cc:654] T b0999fa2827c49be81f591af0a501081 P af25b377ab324b67b36aec602ec84afd: Neither blocks nor log segments found. Creating new log.
I20260811 22:30:18.180805 467 log.cc:824] T b0999fa2827c49be81f591af0a501081 P af25b377ab324b67b36aec602ec84afd: Log is configured to *not* fsync() on all Append() calls
I20260811 22:30:18.182822 468 tablet_bootstrap.cc:492] T b0999fa2827c49be81f591af0a501081 P 78a614bde28849a7b2c0d19b37ad3a12: Bootstrap starting.
I20260811 22:30:18.183650 467 tablet_bootstrap.cc:492] T b0999fa2827c49be81f591af0a501081 P af25b377ab324b67b36aec602ec84afd: No bootstrap required, opened a new log
I20260811 22:30:18.183890 467 ts_tablet_manager.cc:1397] T b0999fa2827c49be81f591af0a501081 P af25b377ab324b67b36aec602ec84afd: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260811 22:30:18.186565 468 tablet_bootstrap.cc:654] T b0999fa2827c49be81f591af0a501081 P 78a614bde28849a7b2c0d19b37ad3a12: Neither blocks nor log segments found. Creating new log.
I20260811 22:30:18.187352 469 tablet_bootstrap.cc:492] T b0999fa2827c49be81f591af0a501081 P 34aa3b0820ae493a8236de282519a94b: Bootstrap starting.
I20260811 22:30:18.187847 468 log.cc:824] T b0999fa2827c49be81f591af0a501081 P 78a614bde28849a7b2c0d19b37ad3a12: Log is configured to *not* fsync() on all Append() calls
I20260811 22:30:18.190513 468 tablet_bootstrap.cc:492] T b0999fa2827c49be81f591af0a501081 P 78a614bde28849a7b2c0d19b37ad3a12: No bootstrap required, opened a new log
I20260811 22:30:18.190766 468 ts_tablet_manager.cc:1397] T b0999fa2827c49be81f591af0a501081 P 78a614bde28849a7b2c0d19b37ad3a12: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260811 22:30:18.191120 469 tablet_bootstrap.cc:654] T b0999fa2827c49be81f591af0a501081 P 34aa3b0820ae493a8236de282519a94b: Neither blocks nor log segments found. Creating new log.
I20260811 22:30:18.192358 469 log.cc:824] T b0999fa2827c49be81f591af0a501081 P 34aa3b0820ae493a8236de282519a94b: Log is configured to *not* fsync() on all Append() calls
I20260811 22:30:18.192791 467 raft_consensus.cc:348] T b0999fa2827c49be81f591af0a501081 P af25b377ab324b67b36aec602ec84afd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a614bde28849a7b2c0d19b37ad3a12" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 46485 } } peers { permanent_uuid: "34aa3b0820ae493a8236de282519a94b" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 44745 } } peers { permanent_uuid: "af25b377ab324b67b36aec602ec84afd" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 36079 } }
I20260811 22:30:18.193216 467 raft_consensus.cc:374] T b0999fa2827c49be81f591af0a501081 P af25b377ab324b67b36aec602ec84afd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:30:18.193315 467 raft_consensus.cc:729] T b0999fa2827c49be81f591af0a501081 P af25b377ab324b67b36aec602ec84afd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af25b377ab324b67b36aec602ec84afd, State: Initialized, Role: FOLLOWER
I20260811 22:30:18.193915 467 consensus_queue.cc:260] T b0999fa2827c49be81f591af0a501081 P af25b377ab324b67b36aec602ec84afd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a614bde28849a7b2c0d19b37ad3a12" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 46485 } } peers { permanent_uuid: "34aa3b0820ae493a8236de282519a94b" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 44745 } } peers { permanent_uuid: "af25b377ab324b67b36aec602ec84afd" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 36079 } }
I20260811 22:30:18.194984 32671 heartbeater.cc:503] Master 127.29.106.62:38057 was elected leader, sending a full tablet report...
I20260811 22:30:18.195451 469 tablet_bootstrap.cc:492] T b0999fa2827c49be81f591af0a501081 P 34aa3b0820ae493a8236de282519a94b: No bootstrap required, opened a new log
I20260811 22:30:18.195732 467 ts_tablet_manager.cc:1428] T b0999fa2827c49be81f591af0a501081 P af25b377ab324b67b36aec602ec84afd: Time spent starting tablet: real 0.012s user 0.004s sys 0.007s
I20260811 22:30:18.196501 469 ts_tablet_manager.cc:1397] T b0999fa2827c49be81f591af0a501081 P 34aa3b0820ae493a8236de282519a94b: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260811 22:30:18.198859 468 raft_consensus.cc:348] T b0999fa2827c49be81f591af0a501081 P 78a614bde28849a7b2c0d19b37ad3a12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a614bde28849a7b2c0d19b37ad3a12" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 46485 } } peers { permanent_uuid: "34aa3b0820ae493a8236de282519a94b" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 44745 } } peers { permanent_uuid: "af25b377ab324b67b36aec602ec84afd" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 36079 } }
I20260811 22:30:18.199273 468 raft_consensus.cc:374] T b0999fa2827c49be81f591af0a501081 P 78a614bde28849a7b2c0d19b37ad3a12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:30:18.199376 468 raft_consensus.cc:729] T b0999fa2827c49be81f591af0a501081 P 78a614bde28849a7b2c0d19b37ad3a12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78a614bde28849a7b2c0d19b37ad3a12, State: Initialized, Role: FOLLOWER
I20260811 22:30:18.199940 468 consensus_queue.cc:260] T b0999fa2827c49be81f591af0a501081 P 78a614bde28849a7b2c0d19b37ad3a12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a614bde28849a7b2c0d19b37ad3a12" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 46485 } } peers { permanent_uuid: "34aa3b0820ae493a8236de282519a94b" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 44745 } } peers { permanent_uuid: "af25b377ab324b67b36aec602ec84afd" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 36079 } }
I20260811 22:30:18.201023 443 heartbeater.cc:503] Master 127.29.106.62:38057 was elected leader, sending a full tablet report...
I20260811 22:30:18.201532 468 ts_tablet_manager.cc:1428] T b0999fa2827c49be81f591af0a501081 P 78a614bde28849a7b2c0d19b37ad3a12: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260811 22:30:18.206868 469 raft_consensus.cc:348] T b0999fa2827c49be81f591af0a501081 P 34aa3b0820ae493a8236de282519a94b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a614bde28849a7b2c0d19b37ad3a12" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 46485 } } peers { permanent_uuid: "34aa3b0820ae493a8236de282519a94b" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 44745 } } peers { permanent_uuid: "af25b377ab324b67b36aec602ec84afd" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 36079 } }
I20260811 22:30:18.207176 469 raft_consensus.cc:374] T b0999fa2827c49be81f591af0a501081 P 34aa3b0820ae493a8236de282519a94b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:30:18.207291 469 raft_consensus.cc:729] T b0999fa2827c49be81f591af0a501081 P 34aa3b0820ae493a8236de282519a94b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34aa3b0820ae493a8236de282519a94b, State: Initialized, Role: FOLLOWER
I20260811 22:30:18.207780 469 consensus_queue.cc:260] T b0999fa2827c49be81f591af0a501081 P 34aa3b0820ae493a8236de282519a94b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a614bde28849a7b2c0d19b37ad3a12" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 46485 } } peers { permanent_uuid: "34aa3b0820ae493a8236de282519a94b" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 44745 } } peers { permanent_uuid: "af25b377ab324b67b36aec602ec84afd" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 36079 } }
I20260811 22:30:18.209000 469 ts_tablet_manager.cc:1428] T b0999fa2827c49be81f591af0a501081 P 34aa3b0820ae493a8236de282519a94b: Time spent starting tablet: real 0.012s user 0.007s sys 0.003s
W20260811 22:30:18.319602 32423 tablet.cc:2369] T b0999fa2827c49be81f591af0a501081 P 34aa3b0820ae493a8236de282519a94b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 22:30:18.328270 445 tablet.cc:2369] T b0999fa2827c49be81f591af0a501081 P 78a614bde28849a7b2c0d19b37ad3a12: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 22:30:18.332437 32672 tablet.cc:2369] T b0999fa2827c49be81f591af0a501081 P af25b377ab324b67b36aec602ec84afd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:30:18.521911 478 raft_consensus.cc:482] T b0999fa2827c49be81f591af0a501081 P 34aa3b0820ae493a8236de282519a94b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:30:18.522219 478 raft_consensus.cc:504] T b0999fa2827c49be81f591af0a501081 P 34aa3b0820ae493a8236de282519a94b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a614bde28849a7b2c0d19b37ad3a12" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 46485 } } peers { permanent_uuid: "34aa3b0820ae493a8236de282519a94b" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 44745 } } peers { permanent_uuid: "af25b377ab324b67b36aec602ec84afd" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 36079 } }
I20260811 22:30:18.523989 478 leader_election.cc:288] T b0999fa2827c49be81f591af0a501081 P 34aa3b0820ae493a8236de282519a94b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78a614bde28849a7b2c0d19b37ad3a12 (127.29.106.3:46485), af25b377ab324b67b36aec602ec84afd (127.29.106.2:36079)
I20260811 22:30:18.530820 466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:18.525201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:60875 (local address 127.29.106.2:36079)
0811 22:30:18.525393 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:18.525399 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:18.525579 (+ 180us) server_negotiation.cc:407] Connection header received
0811 22:30:18.525716 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:18.525721 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:18.525792 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:18.525887 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:30:18.526491 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:18.527229 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:18.529067 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:18.529836 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:18.529974 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:18.530056 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:18.530447 (+ 391us) server_negotiation.cc:299] Negotiation successful
0811 22:30:18.530569 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260811 22:30:18.530850 462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:18.524311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:43595 (local address 127.29.106.3:46485)
0811 22:30:18.524531 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:18.524537 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:18.524949 (+ 412us) server_negotiation.cc:407] Connection header received
0811 22:30:18.525200 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:18.525206 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:18.525285 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:18.525350 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:30:18.526208 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:18.527048 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:18.529007 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:18.529812 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:18.529945 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:18.530017 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:18.530439 (+ 422us) server_negotiation.cc:299] Negotiation successful
0811 22:30:18.530559 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260811 22:30:18.531100 479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:18.524235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.3:46485 (local address 127.29.106.1:43595)
0811 22:30:18.524827 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:18.524847 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:18.525006 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:18.525575 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:18.525579 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:18.525682 (+ 103us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:30:18.526048 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:18.526060 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:18.527181 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:18.527186 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:18.528866 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:18.528878 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:18.530021 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:18.530027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:18.530254 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:18.530316 (+ 62us) client_negotiation.cc:769] Sending connection context
0811 22:30:18.530392 (+ 76us) client_negotiation.cc:240] Negotiation successful
0811 22:30:18.530510 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":116,"spinlock_wait_cycles":24960,"thread_start_us":279,"threads_started":1}
I20260811 22:30:18.531140 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:18.524894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.2:36079 (local address 127.29.106.1:60875)
0811 22:30:18.525485 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:18.525499 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:18.525609 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:18.525901 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:18.525904 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:18.525925 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:30:18.526330 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:18.526343 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:18.527373 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:18.527376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:18.528909 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:18.528921 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:18.530061 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:18.530066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:18.530290 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:18.530324 (+ 34us) client_negotiation.cc:769] Sending connection context
0811 22:30:18.530406 (+ 82us) client_negotiation.cc:240] Negotiation successful
0811 22:30:18.530525 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"mutex_wait_us":65,"thread_start_us":289,"threads_started":1}
I20260811 22:30:18.532377 362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0999fa2827c49be81f591af0a501081" candidate_uuid: "34aa3b0820ae493a8236de282519a94b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78a614bde28849a7b2c0d19b37ad3a12" is_pre_election: true
I20260811 22:30:18.532375 32593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0999fa2827c49be81f591af0a501081" candidate_uuid: "34aa3b0820ae493a8236de282519a94b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af25b377ab324b67b36aec602ec84afd" is_pre_election: true
I20260811 22:30:18.532850 32593 raft_consensus.cc:2454] T b0999fa2827c49be81f591af0a501081 P af25b377ab324b67b36aec602ec84afd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34aa3b0820ae493a8236de282519a94b in term 0.
I20260811 22:30:18.532852 362 raft_consensus.cc:2454] T b0999fa2827c49be81f591af0a501081 P 78a614bde28849a7b2c0d19b37ad3a12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34aa3b0820ae493a8236de282519a94b in term 0.
I20260811 22:30:18.533023 477 raft_consensus.cc:482] T b0999fa2827c49be81f591af0a501081 P 78a614bde28849a7b2c0d19b37ad3a12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:30:18.533229 477 raft_consensus.cc:504] T b0999fa2827c49be81f591af0a501081 P 78a614bde28849a7b2c0d19b37ad3a12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a614bde28849a7b2c0d19b37ad3a12" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 46485 } } peers { permanent_uuid: "34aa3b0820ae493a8236de282519a94b" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 44745 } } peers { permanent_uuid: "af25b377ab324b67b36aec602ec84afd" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 36079 } }
I20260811 22:30:18.533461 32232 leader_election.cc:302] T b0999fa2827c49be81f591af0a501081 P 34aa3b0820ae493a8236de282519a94b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34aa3b0820ae493a8236de282519a94b, af25b377ab324b67b36aec602ec84afd; no voters:
I20260811 22:30:18.533856 478 raft_consensus.cc:2781] T b0999fa2827c49be81f591af0a501081 P 34aa3b0820ae493a8236de282519a94b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:30:18.534027 478 raft_consensus.cc:482] T b0999fa2827c49be81f591af0a501081 P 34aa3b0820ae493a8236de282519a94b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:30:18.534139 478 raft_consensus.cc:3037] T b0999fa2827c49be81f591af0a501081 P 34aa3b0820ae493a8236de282519a94b [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:30:18.534960 477 leader_election.cc:288] T b0999fa2827c49be81f591af0a501081 P 78a614bde28849a7b2c0d19b37ad3a12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34aa3b0820ae493a8236de282519a94b (127.29.106.1:44745), af25b377ab324b67b36aec602ec84afd (127.29.106.2:36079)
I20260811 22:30:18.536805 478 raft_consensus.cc:504] T b0999fa2827c49be81f591af0a501081 P 34aa3b0820ae493a8236de282519a94b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a614bde28849a7b2c0d19b37ad3a12" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 46485 } } peers { permanent_uuid: "34aa3b0820ae493a8236de282519a94b" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 44745 } } peers { permanent_uuid: "af25b377ab324b67b36aec602ec84afd" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 36079 } }
I20260811 22:30:18.537313 478 leader_election.cc:288] T b0999fa2827c49be81f591af0a501081 P 34aa3b0820ae493a8236de282519a94b [CANDIDATE]: Term 1 election: Requested vote from peers 78a614bde28849a7b2c0d19b37ad3a12 (127.29.106.3:46485), af25b377ab324b67b36aec602ec84afd (127.29.106.2:36079)
I20260811 22:30:18.537739 32593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0999fa2827c49be81f591af0a501081" candidate_uuid: "34aa3b0820ae493a8236de282519a94b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af25b377ab324b67b36aec602ec84afd"
I20260811 22:30:18.537909 32593 raft_consensus.cc:3037] T b0999fa2827c49be81f591af0a501081 P af25b377ab324b67b36aec602ec84afd [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:30:18.538028 362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0999fa2827c49be81f591af0a501081" candidate_uuid: "34aa3b0820ae493a8236de282519a94b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78a614bde28849a7b2c0d19b37ad3a12"
I20260811 22:30:18.538220 362 raft_consensus.cc:3037] T b0999fa2827c49be81f591af0a501081 P 78a614bde28849a7b2c0d19b37ad3a12 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:30:18.540182 32593 raft_consensus.cc:2454] T b0999fa2827c49be81f591af0a501081 P af25b377ab324b67b36aec602ec84afd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34aa3b0820ae493a8236de282519a94b in term 1.
I20260811 22:30:18.540527 32232 leader_election.cc:302] T b0999fa2827c49be81f591af0a501081 P 34aa3b0820ae493a8236de282519a94b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34aa3b0820ae493a8236de282519a94b, af25b377ab324b67b36aec602ec84afd; no voters:
I20260811 22:30:18.540767 478 raft_consensus.cc:2781] T b0999fa2827c49be81f591af0a501081 P 34aa3b0820ae493a8236de282519a94b [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:30:18.541184 478 raft_consensus.cc:686] T b0999fa2827c49be81f591af0a501081 P 34aa3b0820ae493a8236de282519a94b [term 1 LEADER]: Becoming Leader. State: Replica: 34aa3b0820ae493a8236de282519a94b, State: Running, Role: LEADER
I20260811 22:30:18.541719 478 consensus_queue.cc:237] T b0999fa2827c49be81f591af0a501081 P 34aa3b0820ae493a8236de282519a94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a614bde28849a7b2c0d19b37ad3a12" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 46485 } } peers { permanent_uuid: "34aa3b0820ae493a8236de282519a94b" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 44745 } } peers { permanent_uuid: "af25b377ab324b67b36aec602ec84afd" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 36079 } }
I20260811 22:30:18.542162 362 raft_consensus.cc:2454] T b0999fa2827c49be81f591af0a501081 P 78a614bde28849a7b2c0d19b37ad3a12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34aa3b0820ae493a8236de282519a94b in term 1.
I20260811 22:30:18.543817 447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:18.535135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:44745 (local address 127.29.106.3:41801)
0811 22:30:18.535328 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:18.535346 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:18.535452 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:18.537802 (+ 2350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:18.537806 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:18.537839 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:30:18.538016 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:18.538027 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:18.539334 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:18.539339 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:18.541303 (+ 1964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:18.541318 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:18.543054 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:18.543059 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:18.543256 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:18.543315 (+ 59us) client_negotiation.cc:769] Sending connection context
0811 22:30:18.543403 (+ 88us) client_negotiation.cc:240] Negotiation successful
0811 22:30:18.543525 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260811 22:30:18.544431 463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:18.535625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:41801 (local address 127.29.106.1:44745)
0811 22:30:18.537567 (+ 1942us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:18.537582 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:18.537598 (+ 16us) server_negotiation.cc:407] Connection header received
0811 22:30:18.537643 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:18.537646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:18.537693 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:18.537769 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:30:18.538192 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:18.539140 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:18.541451 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:18.542680 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:18.542833 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:18.542896 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:18.544044 (+ 1148us) server_negotiation.cc:299] Negotiation successful
0811 22:30:18.544188 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1828}
I20260811 22:30:18.545447 32345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0999fa2827c49be81f591af0a501081" candidate_uuid: "78a614bde28849a7b2c0d19b37ad3a12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34aa3b0820ae493a8236de282519a94b" is_pre_election: true
I20260811 22:30:18.547868 466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:18.535730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:49883 (local address 127.29.106.2:36079)
0811 22:30:18.541728 (+ 5998us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:18.541733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:18.541750 (+ 17us) server_negotiation.cc:407] Connection header received
0811 22:30:18.541798 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:18.541803 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:18.541862 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:18.541953 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:30:18.542588 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:18.543487 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:18.546128 (+ 2641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:18.547016 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:18.547171 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:18.547235 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:18.547512 (+ 277us) server_negotiation.cc:299] Negotiation successful
0811 22:30:18.547626 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5857}
I20260811 22:30:18.547905 485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:18.535336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.2:36079 (local address 127.29.106.3:49883)
0811 22:30:18.536078 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:18.536093 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:18.536178 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:18.541975 (+ 5797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:18.541979 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:18.542003 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:30:18.542421 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:18.542434 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:18.543623 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:18.543628 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:18.545033 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:18.545045 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:18.547196 (+ 2151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:18.547201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:18.547377 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:18.547399 (+ 22us) client_negotiation.cc:769] Sending connection context
0811 22:30:18.547475 (+ 76us) client_negotiation.cc:240] Negotiation successful
0811 22:30:18.547603 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":461,"threads_started":1}
I20260811 22:30:18.548394 32593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0999fa2827c49be81f591af0a501081" candidate_uuid: "78a614bde28849a7b2c0d19b37ad3a12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af25b377ab324b67b36aec602ec84afd" is_pre_election: true
I20260811 22:30:18.548705 32593 raft_consensus.cc:2379] T b0999fa2827c49be81f591af0a501081 P af25b377ab324b67b36aec602ec84afd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78a614bde28849a7b2c0d19b37ad3a12 in current term 1: Already voted for candidate 34aa3b0820ae493a8236de282519a94b in this term.
I20260811 22:30:18.549294 32709 leader_election.cc:302] T b0999fa2827c49be81f591af0a501081 P 78a614bde28849a7b2c0d19b37ad3a12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78a614bde28849a7b2c0d19b37ad3a12; no voters: 34aa3b0820ae493a8236de282519a94b, af25b377ab324b67b36aec602ec84afd
I20260811 22:30:18.549611 477 raft_consensus.cc:2726] T b0999fa2827c49be81f591af0a501081 P 78a614bde28849a7b2c0d19b37ad3a12 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260811 22:30:18.555104 32084 catalog_manager.cc:5495] T b0999fa2827c49be81f591af0a501081 P 34aa3b0820ae493a8236de282519a94b reported cstate change: term changed from 0 to 1, leader changed from <none> to 34aa3b0820ae493a8236de282519a94b (127.29.106.1). New cstate: current_term: 1 leader_uuid: "34aa3b0820ae493a8236de282519a94b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a614bde28849a7b2c0d19b37ad3a12" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 46485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34aa3b0820ae493a8236de282519a94b" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 44745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af25b377ab324b67b36aec602ec84afd" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 36079 } health_report { overall_health: UNKNOWN } } }
I20260811 22:30:18.610945 463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:18.606099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41780 (local address 127.29.106.1:44745)
0811 22:30:18.606282 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:18.606288 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:18.606373 (+ 85us) server_negotiation.cc:407] Connection header received
0811 22:30:18.606526 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:18.606530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:18.606615 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:18.606727 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:30:18.607033 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:18.607791 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:18.609046 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:18.609432 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:18.609526 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:18.610005 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:30:18.610421 (+ 416us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:30:18.610566 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:18.610641 (+ 75us) server_negotiation.cc:299] Negotiation successful
0811 22:30:18.610683 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260811 22:30:18.618366 32593 raft_consensus.cc:1261] T b0999fa2827c49be81f591af0a501081 P af25b377ab324b67b36aec602ec84afd [term 1 FOLLOWER]: Refusing update from remote peer 34aa3b0820ae493a8236de282519a94b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:30:18.618412 362 raft_consensus.cc:1261] T b0999fa2827c49be81f591af0a501081 P 78a614bde28849a7b2c0d19b37ad3a12 [term 1 FOLLOWER]: Refusing update from remote peer 34aa3b0820ae493a8236de282519a94b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:30:18.619194 487 consensus_queue.cc:1035] T b0999fa2827c49be81f591af0a501081 P 34aa3b0820ae493a8236de282519a94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "78a614bde28849a7b2c0d19b37ad3a12" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 46485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:30:18.619598 478 consensus_queue.cc:1035] T b0999fa2827c49be81f591af0a501081 P 34aa3b0820ae493a8236de282519a94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "af25b377ab324b67b36aec602ec84afd" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 36079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:30:18.629966 495 mvcc.cc:205] Tried to move back new op lower bound from 7317452466651983872 to 7317452466351820800. Current Snapshot: MvccSnapshot[applied={T|T < 7317452466651983872}]
I20260811 22:30:18.636215 501 mvcc.cc:205] Tried to move back new op lower bound from 7317452466651983872 to 7317452466351820800. Current Snapshot: MvccSnapshot[applied={T|T < 7317452466651983872}]
I20260811 22:30:18.645300 32085 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:30:18.645521 32085 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:30:18.648860 32085 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cd71674c1d2d43609b8376bd0862f2f5: Sending DeleteTablet for 3 replicas of tablet b0999fa2827c49be81f591af0a501081
I20260811 22:30:18.649808 332 tablet_service.cc:1514] Processing DeleteTablet for tablet b0999fa2827c49be81f591af0a501081 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:30:18 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:41020
I20260811 22:30:18.649955 32310 tablet_service.cc:1514] Processing DeleteTablet for tablet b0999fa2827c49be81f591af0a501081 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:30:18 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:41770
I20260811 22:30:18.650238 32556 tablet_service.cc:1514] Processing DeleteTablet for tablet b0999fa2827c49be81f591af0a501081 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:30:18 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:56650
I20260811 22:30:18.654699 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 32199
I20260811 22:30:18.655730 519 tablet_replica.cc:330] T b0999fa2827c49be81f591af0a501081 P af25b377ab324b67b36aec602ec84afd: stopping tablet replica
I20260811 22:30:18.656138 519 raft_consensus.cc:2229] T b0999fa2827c49be81f591af0a501081 P af25b377ab324b67b36aec602ec84afd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:30:18.656632 519 raft_consensus.cc:2258] T b0999fa2827c49be81f591af0a501081 P af25b377ab324b67b36aec602ec84afd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:30:18.666109 519 ts_tablet_manager.cc:1905] T b0999fa2827c49be81f591af0a501081 P af25b377ab324b67b36aec602ec84afd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:30:18.666561 518 tablet_replica.cc:330] T b0999fa2827c49be81f591af0a501081 P 34aa3b0820ae493a8236de282519a94b: stopping tablet replica
I20260811 22:30:18.666970 518 raft_consensus.cc:2229] T b0999fa2827c49be81f591af0a501081 P 34aa3b0820ae493a8236de282519a94b [term 1 LEADER]: Raft consensus shutting down.
I20260811 22:30:18.667706 518 raft_consensus.cc:2258] T b0999fa2827c49be81f591af0a501081 P 34aa3b0820ae493a8236de282519a94b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:30:18.669703 518 ts_tablet_manager.cc:1905] T b0999fa2827c49be81f591af0a501081 P 34aa3b0820ae493a8236de282519a94b: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 11 22:30:18 dist-test-slave-gjrz krb5kdc[32005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.1@KRBTEST.COM
I20260811 22:30:18.676889 518 ts_tablet_manager.cc:1918] T b0999fa2827c49be81f591af0a501081 P 34aa3b0820ae493a8236de282519a94b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:30:18.677070 518 log.cc:1196] T b0999fa2827c49be81f591af0a501081 P 34aa3b0820ae493a8236de282519a94b: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-0/wal/wals/b0999fa2827c49be81f591af0a501081
I20260811 22:30:18.677630 518 ts_tablet_manager.cc:1939] T b0999fa2827c49be81f591af0a501081 P 34aa3b0820ae493a8236de282519a94b: Deleting consensus metadata
I20260811 22:30:18.678922 517 tablet_replica.cc:330] T b0999fa2827c49be81f591af0a501081 P 78a614bde28849a7b2c0d19b37ad3a12: stopping tablet replica
I20260811 22:30:18.679386 517 raft_consensus.cc:2229] T b0999fa2827c49be81f591af0a501081 P 78a614bde28849a7b2c0d19b37ad3a12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:30:18.679893 517 raft_consensus.cc:2258] T b0999fa2827c49be81f591af0a501081 P 78a614bde28849a7b2c0d19b37ad3a12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:30:18.682086 517 ts_tablet_manager.cc:1905] T b0999fa2827c49be81f591af0a501081 P 78a614bde28849a7b2c0d19b37ad3a12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:30:18.683144 519 ts_tablet_manager.cc:1918] T b0999fa2827c49be81f591af0a501081 P af25b377ab324b67b36aec602ec84afd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:30:18.683341 519 log.cc:1196] T b0999fa2827c49be81f591af0a501081 P af25b377ab324b67b36aec602ec84afd: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-1/wal/wals/b0999fa2827c49be81f591af0a501081
I20260811 22:30:18.683879 519 ts_tablet_manager.cc:1939] T b0999fa2827c49be81f591af0a501081 P af25b377ab324b67b36aec602ec84afd: Deleting consensus metadata
I20260811 22:30:18.684871 463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:18.656006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41794 (local address 127.29.106.1:44745)
0811 22:30:18.656191 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:18.656198 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:18.656491 (+ 293us) server_negotiation.cc:407] Connection header received
0811 22:30:18.656841 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:18.656847 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:18.656950 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:18.657021 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:18.658933 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:18.660202 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:18.662928 (+ 2726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:18.663754 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:18.663852 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:18.677223 (+ 13371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:18.677242 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:18.677263 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:30:18.677307 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:18.680943 (+ 3636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:18.681639 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:18.681648 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:18.681651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:18.681804 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:18.682327 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:18.682332 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:18.682336 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:18.682822 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:18.683117 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:18.684330 (+ 1213us) server_negotiation.cc:299] Negotiation successful
0811 22:30:18.684561 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260811 22:30:18.685702 32058 catalog_manager.cc:4841] TS af25b377ab324b67b36aec602ec84afd (127.29.106.2:36079): tablet b0999fa2827c49be81f591af0a501081 (table test-table [id=3b8fb122709b41a8a527e7079a304e85]) successfully deleted
I20260811 22:30:18.686550 32412 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:30:18.687285 32055 catalog_manager.cc:4841] TS 34aa3b0820ae493a8236de282519a94b (127.29.106.1:44745): tablet b0999fa2827c49be81f591af0a501081 (table test-table [id=3b8fb122709b41a8a527e7079a304e85]) successfully deleted
I20260811 22:30:18.689226 517 ts_tablet_manager.cc:1918] T b0999fa2827c49be81f591af0a501081 P 78a614bde28849a7b2c0d19b37ad3a12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:30:18.689368 517 log.cc:1196] T b0999fa2827c49be81f591af0a501081 P 78a614bde28849a7b2c0d19b37ad3a12: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786487105757681-30120-0/minicluster-data/ts-2/wal/wals/b0999fa2827c49be81f591af0a501081
I20260811 22:30:18.689826 517 ts_tablet_manager.cc:1939] T b0999fa2827c49be81f591af0a501081 P 78a614bde28849a7b2c0d19b37ad3a12: Deleting consensus metadata
I20260811 22:30:18.691372 32058 catalog_manager.cc:4841] TS 78a614bde28849a7b2c0d19b37ad3a12 (127.29.106.3:46485): tablet b0999fa2827c49be81f591af0a501081 (table test-table [id=3b8fb122709b41a8a527e7079a304e85]) successfully deleted
I20260811 22:30:18.783702 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 32199
I20260811 22:30:18.808568 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 32431
Aug 11 22:30:18 dist-test-slave-gjrz krb5kdc[32005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.2@KRBTEST.COM
I20260811 22:30:18.822901 466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:18.809293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56666 (local address 127.29.106.2:36079)
0811 22:30:18.809462 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:18.809468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:18.809538 (+ 70us) server_negotiation.cc:407] Connection header received
0811 22:30:18.809753 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:18.809757 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:18.809823 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:18.809906 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:18.811174 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:18.811907 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:18.813140 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:18.813550 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:18.813663 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:18.817964 (+ 4301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:18.817977 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:18.818010 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:30:18.818047 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:18.820702 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:18.821302 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:18.821308 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:18.821310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:18.821440 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:18.821716 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:18.821719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:18.821721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:18.822054 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:18.822276 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:18.822465 (+ 189us) server_negotiation.cc:299] Negotiation successful
0811 22:30:18.822641 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260811 22:30:18.823511 32659 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:30:18.909811 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 32431
I20260811 22:30:18.937294 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 32680
Aug 11 22:30:18 dist-test-slave-gjrz krb5kdc[32005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.3@KRBTEST.COM
I20260811 22:30:18.951582 462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:18.938049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41036 (local address 127.29.106.3:46485)
0811 22:30:18.938228 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:18.938232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:18.938252 (+ 20us) server_negotiation.cc:407] Connection header received
0811 22:30:18.938364 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:18.938368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:18.938453 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:18.938538 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:18.939817 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:18.940559 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:18.941809 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:18.942218 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:18.942312 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:18.946631 (+ 4319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:18.946647 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:18.946665 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:30:18.946697 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:18.949290 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:18.949818 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:18.949826 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:18.949828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:18.949973 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:18.950266 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:18.950269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:18.950271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:18.950638 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:18.950848 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:18.951062 (+ 214us) server_negotiation.cc:299] Negotiation successful
0811 22:30:18.951216 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260811 22:30:18.952165 431 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:30:19.040103 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 32680
I20260811 22:30:19.065128 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 32025
I20260811 22:30:19.066181 32139 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:30:19.176189 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 32025
2026-08-11T22:30:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4721 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-11T22:30:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:30:19Z Disabled control of system clock
I20260811 22:30:19.269085 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:44179
--webserver_interface=127.29.106.62
--webserver_port=0
--builtin_ntp_servers=127.29.106.20:42535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:44179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 22:30:19.599236 536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:30:19.599538 536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:30:19.599664 536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:30:19.606734 536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:30:19.606830 536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:30:19.606889 536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:30:19.606968 536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:30:19.607035 536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:30:19.615967 536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:42535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:44179
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:44179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.106.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:30:19.617177 536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:30:19.618999 536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:30:19.630828 548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:19.630789 555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:19.630874 551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:30:19.631929 536 server_base.cc:1034] running on GCE node
I20260811 22:30:19.632663 536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:30:19.633941 536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:30:19.635154 536 hybrid_clock.cc:648] HybridClock initialized: now 1786487419635099 us; error 46 us; skew 500 ppm
I20260811 22:30:19.635516 536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 22:30:19.637701 536 webserver.cc:460] Webserver started at http://127.29.106.62:34159/ using document root <none> and password file <none>
I20260811 22:30:19.638335 536 fs_manager.cc:359] Metadata directory not provided
I20260811 22:30:19.638437 536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:30:19.638734 536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:30:19.641572 536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "d8226669e18f4bca920666d4c0c60a69"
format_stamp: "Formatted at 2026-08-11 22:30:19 on dist-test-slave-gjrz"
server_key: "c51f34603f1b22298482c0fa5cb2203e"
server_key_iv: "cd09caf6375109a589bf67666efb9519"
server_key_version: "encryptionkey@0"
I20260811 22:30:19.642351 536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "d8226669e18f4bca920666d4c0c60a69"
format_stamp: "Formatted at 2026-08-11 22:30:19 on dist-test-slave-gjrz"
server_key: "c51f34603f1b22298482c0fa5cb2203e"
server_key_iv: "cd09caf6375109a589bf67666efb9519"
server_key_version: "encryptionkey@0"
I20260811 22:30:19.647307 536 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260811 22:30:19.650590 565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:19.651811 536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260811 22:30:19.651963 536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "d8226669e18f4bca920666d4c0c60a69"
format_stamp: "Formatted at 2026-08-11 22:30:19 on dist-test-slave-gjrz"
server_key: "c51f34603f1b22298482c0fa5cb2203e"
server_key_iv: "cd09caf6375109a589bf67666efb9519"
server_key_version: "encryptionkey@0"
I20260811 22:30:19.652163 536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:30:19.675522 536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:30:19.676304 536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:30:19.676582 536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:30:19.694345 536 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.62:44179
I20260811 22:30:19.694365 661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.62:44179 every 8 connection(s)
I20260811 22:30:19.695622 536 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:30:19.699172 662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:30:19.701119 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 536
I20260811 22:30:19.701287 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:30:19.701725 30120 external_mini_cluster.cc:1428] Setting key ef351e4a15310803aea8ead076980a14
I20260811 22:30:19.712364 662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8226669e18f4bca920666d4c0c60a69: Bootstrap starting.
I20260811 22:30:19.716296 662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8226669e18f4bca920666d4c0c60a69: Neither blocks nor log segments found. Creating new log.
I20260811 22:30:19.717232 668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:19.703649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43932 (local address 127.29.106.62:44179)
0811 22:30:19.704725 (+ 1076us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:19.704744 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:19.704798 (+ 54us) server_negotiation.cc:407] Connection header received
0811 22:30:19.707151 (+ 2353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:19.707180 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:19.707780 (+ 600us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:19.708215 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:19.708795 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:19.710800 (+ 2005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:19.712363 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:19.713362 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:19.713572 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:19.714562 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:19.714573 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:19.714591 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:30:19.715251 (+ 660us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:19.715396 (+ 145us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:30:19.715425 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:19.715556 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:19.715694 (+ 138us) server_negotiation.cc:299] Negotiation successful
0811 22:30:19.715995 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":337,"threads_started":1}
I20260811 22:30:19.717641 662 log.cc:824] T 00000000000000000000000000000000 P d8226669e18f4bca920666d4c0c60a69: Log is configured to *not* fsync() on all Append() calls
I20260811 22:30:19.720705 662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8226669e18f4bca920666d4c0c60a69: No bootstrap required, opened a new log
I20260811 22:30:19.728436 662 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8226669e18f4bca920666d4c0c60a69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8226669e18f4bca920666d4c0c60a69" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 44179 } }
I20260811 22:30:19.728768 662 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8226669e18f4bca920666d4c0c60a69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:30:19.728865 662 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8226669e18f4bca920666d4c0c60a69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8226669e18f4bca920666d4c0c60a69, State: Initialized, Role: FOLLOWER
I20260811 22:30:19.729373 662 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8226669e18f4bca920666d4c0c60a69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8226669e18f4bca920666d4c0c60a69" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 44179 } }
I20260811 22:30:19.729593 662 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8226669e18f4bca920666d4c0c60a69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:30:19.729693 662 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8226669e18f4bca920666d4c0c60a69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:30:19.729805 662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8226669e18f4bca920666d4c0c60a69 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:30:19.732152 662 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8226669e18f4bca920666d4c0c60a69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8226669e18f4bca920666d4c0c60a69" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 44179 } }
I20260811 22:30:19.732576 662 leader_election.cc:302] T 00000000000000000000000000000000 P d8226669e18f4bca920666d4c0c60a69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8226669e18f4bca920666d4c0c60a69; no voters:
I20260811 22:30:19.733137 662 leader_election.cc:288] T 00000000000000000000000000000000 P d8226669e18f4bca920666d4c0c60a69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:30:19.733282 673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8226669e18f4bca920666d4c0c60a69 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:30:19.734030 673 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8226669e18f4bca920666d4c0c60a69 [term 1 LEADER]: Becoming Leader. State: Replica: d8226669e18f4bca920666d4c0c60a69, State: Running, Role: LEADER
I20260811 22:30:19.734566 673 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8226669e18f4bca920666d4c0c60a69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8226669e18f4bca920666d4c0c60a69" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 44179 } }
I20260811 22:30:19.735050 662 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8226669e18f4bca920666d4c0c60a69 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:30:19.737668 678 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8226669e18f4bca920666d4c0c60a69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8226669e18f4bca920666d4c0c60a69. Latest consensus state: current_term: 1 leader_uuid: "d8226669e18f4bca920666d4c0c60a69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8226669e18f4bca920666d4c0c60a69" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 44179 } } }
I20260811 22:30:19.738426 678 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8226669e18f4bca920666d4c0c60a69 [sys.catalog]: This master's current role is: LEADER
I20260811 22:30:19.739111 674 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8226669e18f4bca920666d4c0c60a69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8226669e18f4bca920666d4c0c60a69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8226669e18f4bca920666d4c0c60a69" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 44179 } } }
I20260811 22:30:19.739461 696 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:30:19.739481 674 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8226669e18f4bca920666d4c0c60a69 [sys.catalog]: This master's current role is: LEADER
I20260811 22:30:19.744408 696 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:30:19.752017 696 catalog_manager.cc:1347] Generated new cluster ID: 5fa1a99810fb49ab9815e10744a88a76
I20260811 22:30:19.752115 696 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:30:19.769342 696 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:30:19.770443 696 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:30:19.778275 696 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8226669e18f4bca920666d4c0c60a69: Generated new TSK 0
I20260811 22:30:19.779115 696 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:30:19.791745 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.1:0
--local_ip_for_outbound_sockets=127.29.106.1
--webserver_interface=127.29.106.1
--webserver_port=0
--tserver_master_addrs=127.29.106.62:44179
--builtin_ntp_servers=127.29.106.20:42535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 22:30:20.122927 715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:30:20.123183 715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:30:20.123314 715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:30:20.130460 715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:30:20.130548 715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:30:20.130687 715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.1
I20260811 22:30:20.139853 715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:42535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.106.1
--webserver_port=0
--tserver_master_addrs=127.29.106.62:44179
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.1
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:30:20.141063 715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:30:20.143028 715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:30:20.155385 723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:20.155423 725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:20.155452 728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:30:20.156672 715 server_base.cc:1034] running on GCE node
I20260811 22:30:20.157199 715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:30:20.158021 715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:30:20.159212 715 hybrid_clock.cc:648] HybridClock initialized: now 1786487420159161 us; error 45 us; skew 500 ppm
I20260811 22:30:20.159514 715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 22:30:20.161583 715 webserver.cc:460] Webserver started at http://127.29.106.1:35629/ using document root <none> and password file <none>
I20260811 22:30:20.162294 715 fs_manager.cc:359] Metadata directory not provided
I20260811 22:30:20.162384 715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:30:20.162679 715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:30:20.165386 715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-0/data/instance:
uuid: "a03b794ad15347dba5c0880781152c23"
format_stamp: "Formatted at 2026-08-11 22:30:20 on dist-test-slave-gjrz"
server_key: "673c7eedaff51d5e8b3643228b79f2b6"
server_key_iv: "108b55d2263f69fff7361c197ec54167"
server_key_version: "encryptionkey@0"
I20260811 22:30:20.166136 715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance:
uuid: "a03b794ad15347dba5c0880781152c23"
format_stamp: "Formatted at 2026-08-11 22:30:20 on dist-test-slave-gjrz"
server_key: "673c7eedaff51d5e8b3643228b79f2b6"
server_key_iv: "108b55d2263f69fff7361c197ec54167"
server_key_version: "encryptionkey@0"
I20260811 22:30:20.170822 715 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260811 22:30:20.174073 741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:20.175236 715 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 22:30:20.175441 715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-0/wal
uuid: "a03b794ad15347dba5c0880781152c23"
format_stamp: "Formatted at 2026-08-11 22:30:20 on dist-test-slave-gjrz"
server_key: "673c7eedaff51d5e8b3643228b79f2b6"
server_key_iv: "108b55d2263f69fff7361c197ec54167"
server_key_version: "encryptionkey@0"
I20260811 22:30:20.175628 715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:30:20.198365 715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:30:20.199116 715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:30:20.199419 715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:30:20.200312 715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:30:20.201714 715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:30:20.201805 715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:20.201920 715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:30:20.202001 715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:20.232323 715 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.1:38975
I20260811 22:30:20.232393 953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.1:38975 every 8 connection(s)
I20260811 22:30:20.233511 715 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
I20260811 22:30:20.243064 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 715
I20260811 22:30:20.243227 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance
I20260811 22:30:20.243767 30120 external_mini_cluster.cc:1428] Setting key 4d1654c785df3774a11c6908a153d89c
I20260811 22:30:20.245934 962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:20.235969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:53649 (local address 127.29.106.62:44179)
0811 22:30:20.236422 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:20.236429 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:20.237633 (+ 1204us) server_negotiation.cc:407] Connection header received
0811 22:30:20.239563 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:20.239567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:20.239635 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:20.239718 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:20.240738 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:20.241418 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:20.242953 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:20.243447 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:20.243554 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:20.245060 (+ 1506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:20.245067 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:20.245071 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:30:20.245092 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:20.245164 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:30:20.245180 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:20.245295 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:20.245519 (+ 224us) server_negotiation.cc:299] Negotiation successful
0811 22:30:20.245625 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":206,"threads_started":1}
I20260811 22:30:20.246812 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.2:0
--local_ip_for_outbound_sockets=127.29.106.2
--webserver_interface=127.29.106.2
--webserver_port=0
--tserver_master_addrs=127.29.106.62:44179
--builtin_ntp_servers=127.29.106.20:42535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260811 22:30:20.247066 963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:20.236596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:44179 (local address 127.29.106.1:53649)
0811 22:30:20.237479 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:20.237524 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:20.239242 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:20.239967 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:20.239975 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:20.240071 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:30:20.240602 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:20.240618 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:20.241538 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:20.241541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:20.242831 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:20.242847 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:20.243674 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:20.243683 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:20.243969 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:20.244740 (+ 771us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 22:30:20.244743 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:30:20.244791 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 22:30:20.244869 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:30:20.245300 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:30:20.245311 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:30:20.245317 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 22:30:20.245480 (+ 163us) client_negotiation.cc:240] Negotiation successful
0811 22:30:20.245823 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":257,"threads_started":1}
I20260811 22:30:20.248626 954 heartbeater.cc:347] Connected to a master server at 127.29.106.62:44179
I20260811 22:30:20.248883 954 heartbeater.cc:464] Registering TS with master...
I20260811 22:30:20.249532 954 heartbeater.cc:511] Master 127.29.106.62:44179 requested a full tablet report, sending...
I20260811 22:30:20.251636 594 ts_manager.cc:194] Registered new tserver with Master: a03b794ad15347dba5c0880781152c23 (127.29.106.1:38975)
I20260811 22:30:20.253454 594 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.106.1:53649
W20260811 22:30:20.570290 965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:30:20.570524 965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:30:20.570628 965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:30:20.577535 965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:30:20.577616 965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:30:20.577741 965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.2
I20260811 22:30:20.586315 965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:42535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.106.2
--webserver_port=0
--tserver_master_addrs=127.29.106.62:44179
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.2
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:30:20.587422 965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:30:20.589032 965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:30:20.600931 972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:20.600962 971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:20.600903 976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:30:20.601740 965 server_base.cc:1034] running on GCE node
I20260811 22:30:20.602280 965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:30:20.603118 965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:30:20.604321 965 hybrid_clock.cc:648] HybridClock initialized: now 1786487420604262 us; error 43 us; skew 500 ppm
I20260811 22:30:20.604606 965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 22:30:20.606647 965 webserver.cc:460] Webserver started at http://127.29.106.2:40901/ using document root <none> and password file <none>
I20260811 22:30:20.607254 965 fs_manager.cc:359] Metadata directory not provided
I20260811 22:30:20.607391 965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:30:20.607679 965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:30:20.610313 965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-1/data/instance:
uuid: "0480f5bf6b9c47b49eeccd5da15b60e4"
format_stamp: "Formatted at 2026-08-11 22:30:20 on dist-test-slave-gjrz"
server_key: "dd7f8893a97b2f911785f45199c046e4"
server_key_iv: "0948c7b713570bae1fb56695fa8e71dd"
server_key_version: "encryptionkey@0"
I20260811 22:30:20.611027 965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance:
uuid: "0480f5bf6b9c47b49eeccd5da15b60e4"
format_stamp: "Formatted at 2026-08-11 22:30:20 on dist-test-slave-gjrz"
server_key: "dd7f8893a97b2f911785f45199c046e4"
server_key_iv: "0948c7b713570bae1fb56695fa8e71dd"
server_key_version: "encryptionkey@0"
I20260811 22:30:20.615648 965 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260811 22:30:20.618765 993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:20.619939 965 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:30:20.620085 965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-1/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-1/wal
uuid: "0480f5bf6b9c47b49eeccd5da15b60e4"
format_stamp: "Formatted at 2026-08-11 22:30:20 on dist-test-slave-gjrz"
server_key: "dd7f8893a97b2f911785f45199c046e4"
server_key_iv: "0948c7b713570bae1fb56695fa8e71dd"
server_key_version: "encryptionkey@0"
I20260811 22:30:20.620270 965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:30:20.634476 965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:30:20.635165 965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:30:20.635440 965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:30:20.636320 965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:30:20.637660 965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:30:20.637753 965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:20.637856 965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:30:20.637925 965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:20.667248 965 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.2:40067
I20260811 22:30:20.667333 1217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.2:40067 every 8 connection(s)
I20260811 22:30:20.668448 965 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
I20260811 22:30:20.674853 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 965
I20260811 22:30:20.675051 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance
I20260811 22:30:20.675519 30120 external_mini_cluster.cc:1428] Setting key f755a2b9835105bb3dafde7bb3ea6cce
I20260811 22:30:20.678192 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.3:0
--local_ip_for_outbound_sockets=127.29.106.3
--webserver_interface=127.29.106.3
--webserver_port=0
--tserver_master_addrs=127.29.106.62:44179
--builtin_ntp_servers=127.29.106.20:42535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260811 22:30:20.680320 962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:20.670781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:40069 (local address 127.29.106.62:44179)
0811 22:30:20.670943 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:20.670948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:20.672279 (+ 1331us) server_negotiation.cc:407] Connection header received
0811 22:30:20.673937 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:20.673941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:20.674009 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:20.674095 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:20.675062 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:20.675896 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:20.677620 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:20.678026 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:20.678144 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:20.679539 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:20.679546 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:20.679551 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:30:20.679568 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:20.679599 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:30:20.679610 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:20.679683 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:20.679900 (+ 217us) server_negotiation.cc:299] Negotiation successful
0811 22:30:20.679998 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260811 22:30:20.681229 1222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:20.671297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:44179 (local address 127.29.106.2:40069)
0811 22:30:20.672137 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:20.672176 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:20.673687 (+ 1511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:20.674265 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:20.674272 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:20.674349 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:30:20.674877 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:20.674894 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:20.676054 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:20.676058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:20.677479 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:20.677492 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:20.678237 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:20.678244 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:20.678575 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:20.679211 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 22:30:20.679213 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:30:20.679247 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 22:30:20.679346 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:30:20.679693 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:30:20.679705 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:30:20.679713 (+ 8us) client_negotiation.cc:769] Sending connection context
0811 22:30:20.679875 (+ 162us) client_negotiation.cc:240] Negotiation successful
0811 22:30:20.680220 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":265,"threads_started":1}
I20260811 22:30:20.682422 1219 heartbeater.cc:347] Connected to a master server at 127.29.106.62:44179
I20260811 22:30:20.682672 1219 heartbeater.cc:464] Registering TS with master...
I20260811 22:30:20.683406 1219 heartbeater.cc:511] Master 127.29.106.62:44179 requested a full tablet report, sending...
I20260811 22:30:20.684809 594 ts_manager.cc:194] Registered new tserver with Master: 0480f5bf6b9c47b49eeccd5da15b60e4 (127.29.106.2:40067)
I20260811 22:30:20.685791 594 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.106.2:40069
W20260811 22:30:21.008440 1224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:30:21.008687 1224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:30:21.008775 1224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:30:21.015764 1224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:30:21.015852 1224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:30:21.015962 1224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.3
I20260811 22:30:21.024688 1224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:42535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.106.3
--webserver_port=0
--tserver_master_addrs=127.29.106.62:44179
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.3
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:30:21.025781 1224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:30:21.027546 1224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:30:21.039971 1232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:21.039988 1233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:21.040033 1236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:30:21.041036 1224 server_base.cc:1034] running on GCE node
I20260811 22:30:21.041543 1224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:30:21.042380 1224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:30:21.043579 1224 hybrid_clock.cc:648] HybridClock initialized: now 1786487421043525 us; error 44 us; skew 500 ppm
I20260811 22:30:21.043862 1224 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 22:30:21.045934 1224 webserver.cc:460] Webserver started at http://127.29.106.3:43731/ using document root <none> and password file <none>
I20260811 22:30:21.046536 1224 fs_manager.cc:359] Metadata directory not provided
I20260811 22:30:21.046625 1224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:30:21.046932 1224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:30:21.049703 1224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-2/data/instance:
uuid: "4ad4667f717e4280bd2a6872f7701f8f"
format_stamp: "Formatted at 2026-08-11 22:30:21 on dist-test-slave-gjrz"
server_key: "0e8cf84a3f1378b1d79a6b30ab33172b"
server_key_iv: "c28746dd57d90dc7319e5581943858d0"
server_key_version: "encryptionkey@0"
I20260811 22:30:21.050467 1224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance:
uuid: "4ad4667f717e4280bd2a6872f7701f8f"
format_stamp: "Formatted at 2026-08-11 22:30:21 on dist-test-slave-gjrz"
server_key: "0e8cf84a3f1378b1d79a6b30ab33172b"
server_key_iv: "c28746dd57d90dc7319e5581943858d0"
server_key_version: "encryptionkey@0"
I20260811 22:30:21.055250 1224 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260811 22:30:21.058601 1247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:21.059878 1224 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260811 22:30:21.060035 1224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-2/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-2/wal
uuid: "4ad4667f717e4280bd2a6872f7701f8f"
format_stamp: "Formatted at 2026-08-11 22:30:21 on dist-test-slave-gjrz"
server_key: "0e8cf84a3f1378b1d79a6b30ab33172b"
server_key_iv: "c28746dd57d90dc7319e5581943858d0"
server_key_version: "encryptionkey@0"
I20260811 22:30:21.060222 1224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:30:21.078312 1224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:30:21.079010 1224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:30:21.079289 1224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:30:21.080168 1224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:30:21.081471 1224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:30:21.081569 1224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:21.081679 1224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:30:21.081756 1224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:21.111680 1224 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.3:36809
I20260811 22:30:21.111739 1475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.3:36809 every 8 connection(s)
I20260811 22:30:21.112910 1224 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
I20260811 22:30:21.117738 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 1224
I20260811 22:30:21.117882 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance
I20260811 22:30:21.118292 30120 external_mini_cluster.cc:1428] Setting key 24a6d2601539529bfdb0411a81193d01
I20260811 22:30:21.124769 962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:21.115175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:39051 (local address 127.29.106.62:44179)
0811 22:30:21.115427 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:21.115434 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:21.116718 (+ 1284us) server_negotiation.cc:407] Connection header received
0811 22:30:21.118559 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:21.118564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:21.118652 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:21.118764 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:21.119927 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:21.120682 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:21.122200 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:21.122600 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:21.122710 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:21.123932 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:21.123937 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:21.123940 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:30:21.123957 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:21.123988 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:30:21.124021 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:21.124101 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:21.124328 (+ 227us) server_negotiation.cc:299] Negotiation successful
0811 22:30:21.124450 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260811 22:30:21.125622 1483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:21.115709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:44179 (local address 127.29.106.3:39051)
0811 22:30:21.116567 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:21.116609 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:21.118271 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:21.119018 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:21.119026 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:21.119107 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:30:21.119759 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:21.119777 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:21.120831 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:21.120835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:21.122062 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:21.122074 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:21.122808 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:21.122816 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:21.123020 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:21.123619 (+ 599us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 22:30:21.123622 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:30:21.123660 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 22:30:21.123750 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:30:21.124117 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:30:21.124129 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:30:21.124137 (+ 8us) client_negotiation.cc:769] Sending connection context
0811 22:30:21.124279 (+ 142us) client_negotiation.cc:240] Negotiation successful
0811 22:30:21.124660 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":259,"threads_started":1}
I20260811 22:30:21.126744 1478 heartbeater.cc:347] Connected to a master server at 127.29.106.62:44179
I20260811 22:30:21.127015 1478 heartbeater.cc:464] Registering TS with master...
I20260811 22:30:21.127736 1478 heartbeater.cc:511] Master 127.29.106.62:44179 requested a full tablet report, sending...
I20260811 22:30:21.128942 594 ts_manager.cc:194] Registered new tserver with Master: 4ad4667f717e4280bd2a6872f7701f8f (127.29.106.3:36809)
I20260811 22:30:21.129712 594 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.106.3:39051
I20260811 22:30:21.133111 30120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 22:30:21.142604 962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:21.136946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43940 (local address 127.29.106.62:44179)
0811 22:30:21.137087 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:21.137091 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:21.137502 (+ 411us) server_negotiation.cc:407] Connection header received
0811 22:30:21.137651 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:21.137655 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:21.137708 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:21.137788 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:21.138275 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:21.139000 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:21.140392 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:21.140770 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:21.140881 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:21.141942 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:21.141947 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:21.141950 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:30:21.141971 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:21.142000 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:30:21.142011 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:21.142137 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:21.142318 (+ 181us) server_negotiation.cc:299] Negotiation successful
0811 22:30:21.142372 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 22:30:21.145960 594 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43940:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:30:21.152635 594 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:30:21.191164 1515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:21.165029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.3:36809 (local address 127.0.0.1:53384)
0811 22:30:21.177621 (+ 12592us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:21.177672 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:21.177849 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:21.183294 (+ 5445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:21.183302 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:21.183340 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:30:21.184521 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:21.184537 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:21.186478 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:21.186484 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:21.188609 (+ 2125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:21.188626 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:21.190322 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:21.190329 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:21.190519 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:21.190571 (+ 52us) client_negotiation.cc:769] Sending connection context
0811 22:30:21.190680 (+ 109us) client_negotiation.cc:240] Negotiation successful
0811 22:30:21.190855 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12317,"thread_start_us":12173,"threads_started":1}
I20260811 22:30:21.191244 1520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:21.165557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53384 (local address 127.29.106.3:36809)
0811 22:30:21.182634 (+ 17077us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:21.182644 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:21.182684 (+ 40us) server_negotiation.cc:407] Connection header received
0811 22:30:21.182786 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:21.182794 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:21.183076 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:21.183285 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:30:21.184726 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:21.186361 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:21.188836 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:21.190139 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:21.190259 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:21.190553 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:21.190753 (+ 200us) server_negotiation.cc:299] Negotiation successful
0811 22:30:21.190909 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16889,"thread_start_us":18332,"threads_started":1}
I20260811 22:30:21.192692 1522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:21.165980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48324 (local address 127.29.106.2:40067)
0811 22:30:21.183877 (+ 17897us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:21.183885 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:21.185222 (+ 1337us) server_negotiation.cc:407] Connection header received
0811 22:30:21.185318 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:21.185323 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:21.185481 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:21.185699 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:30:21.186176 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:21.187568 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:21.189875 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:21.191310 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:21.191413 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:21.191629 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:21.192263 (+ 634us) server_negotiation.cc:299] Negotiation successful
0811 22:30:21.192420 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17748,"thread_start_us":17635,"threads_started":1}
I20260811 22:30:21.192719 1519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:21.165138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:38975 (local address 127.0.0.1:45462)
0811 22:30:21.181913 (+ 16775us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:21.181934 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:21.182073 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:21.185018 (+ 2945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:21.185023 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:21.185051 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:30:21.185443 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:21.185457 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:21.187222 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:21.187227 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:21.189624 (+ 2397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:21.189639 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:21.191820 (+ 2181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:21.191827 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:21.192043 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:21.192099 (+ 56us) client_negotiation.cc:769] Sending connection context
0811 22:30:21.192188 (+ 89us) client_negotiation.cc:240] Negotiation successful
0811 22:30:21.192376 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16627,"mutex_wait_us":88,"thread_start_us":18948,"threads_started":1}
I20260811 22:30:21.193397 1518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:21.165266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45462 (local address 127.29.106.1:38975)
0811 22:30:21.184458 (+ 19192us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:21.184466 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:21.184498 (+ 32us) server_negotiation.cc:407] Connection header received
0811 22:30:21.184592 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:21.184599 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:21.184823 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:21.185027 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:30:21.185641 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:21.187073 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:21.189838 (+ 2765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:21.191253 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:21.191410 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:21.191629 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:21.192915 (+ 1286us) server_negotiation.cc:299] Negotiation successful
0811 22:30:21.193084 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19033,"thread_start_us":18912,"threads_started":1}
I20260811 22:30:21.195169 1354 tablet_service.cc:1467] Processing CreateTablet for tablet 3104e9581b7544e4880e3bae741abb34 (DEFAULT_TABLE table=test-table [id=a277099c39fb4d1788c3d1a848ba2665]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:30:21.196532 1516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:21.165341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.2:40067 (local address 127.0.0.1:48324)
0811 22:30:21.184955 (+ 19614us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:21.184972 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:21.185088 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:21.185662 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:21.185667 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:21.185692 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:30:21.186019 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:21.186031 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:21.187700 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:21.187706 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:21.189629 (+ 1923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:21.189642 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:21.191772 (+ 2130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:21.191779 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:21.192028 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:21.192095 (+ 67us) client_negotiation.cc:769] Sending connection context
0811 22:30:21.192188 (+ 93us) client_negotiation.cc:240] Negotiation successful
0811 22:30:21.196152 (+ 3964us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19413,"mutex_wait_us":114,"spinlock_wait_cycles":2432,"thread_start_us":16232,"threads_started":1}
I20260811 22:30:21.196568 1354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3104e9581b7544e4880e3bae741abb34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:30:21.198235 837 tablet_service.cc:1467] Processing CreateTablet for tablet 3104e9581b7544e4880e3bae741abb34 (DEFAULT_TABLE table=test-table [id=a277099c39fb4d1788c3d1a848ba2665]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:30:21.199409 837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3104e9581b7544e4880e3bae741abb34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:30:21.201800 1096 tablet_service.cc:1467] Processing CreateTablet for tablet 3104e9581b7544e4880e3bae741abb34 (DEFAULT_TABLE table=test-table [id=a277099c39fb4d1788c3d1a848ba2665]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:30:21.202790 1096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3104e9581b7544e4880e3bae741abb34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:30:21.209350 1525 tablet_bootstrap.cc:492] T 3104e9581b7544e4880e3bae741abb34 P 4ad4667f717e4280bd2a6872f7701f8f: Bootstrap starting.
I20260811 22:30:21.213002 1525 tablet_bootstrap.cc:654] T 3104e9581b7544e4880e3bae741abb34 P 4ad4667f717e4280bd2a6872f7701f8f: Neither blocks nor log segments found. Creating new log.
I20260811 22:30:21.214363 1525 log.cc:824] T 3104e9581b7544e4880e3bae741abb34 P 4ad4667f717e4280bd2a6872f7701f8f: Log is configured to *not* fsync() on all Append() calls
I20260811 22:30:21.214458 1529 tablet_bootstrap.cc:492] T 3104e9581b7544e4880e3bae741abb34 P a03b794ad15347dba5c0880781152c23: Bootstrap starting.
I20260811 22:30:21.215885 1531 tablet_bootstrap.cc:492] T 3104e9581b7544e4880e3bae741abb34 P 0480f5bf6b9c47b49eeccd5da15b60e4: Bootstrap starting.
I20260811 22:30:21.218006 1529 tablet_bootstrap.cc:654] T 3104e9581b7544e4880e3bae741abb34 P a03b794ad15347dba5c0880781152c23: Neither blocks nor log segments found. Creating new log.
I20260811 22:30:21.218688 1525 tablet_bootstrap.cc:492] T 3104e9581b7544e4880e3bae741abb34 P 4ad4667f717e4280bd2a6872f7701f8f: No bootstrap required, opened a new log
I20260811 22:30:21.218927 1525 ts_tablet_manager.cc:1397] T 3104e9581b7544e4880e3bae741abb34 P 4ad4667f717e4280bd2a6872f7701f8f: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260811 22:30:21.219221 1531 tablet_bootstrap.cc:654] T 3104e9581b7544e4880e3bae741abb34 P 0480f5bf6b9c47b49eeccd5da15b60e4: Neither blocks nor log segments found. Creating new log.
I20260811 22:30:21.219205 1529 log.cc:824] T 3104e9581b7544e4880e3bae741abb34 P a03b794ad15347dba5c0880781152c23: Log is configured to *not* fsync() on all Append() calls
I20260811 22:30:21.220306 1531 log.cc:824] T 3104e9581b7544e4880e3bae741abb34 P 0480f5bf6b9c47b49eeccd5da15b60e4: Log is configured to *not* fsync() on all Append() calls
I20260811 22:30:21.222110 1529 tablet_bootstrap.cc:492] T 3104e9581b7544e4880e3bae741abb34 P a03b794ad15347dba5c0880781152c23: No bootstrap required, opened a new log
I20260811 22:30:21.222378 1529 ts_tablet_manager.cc:1397] T 3104e9581b7544e4880e3bae741abb34 P a03b794ad15347dba5c0880781152c23: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260811 22:30:21.223168 1531 tablet_bootstrap.cc:492] T 3104e9581b7544e4880e3bae741abb34 P 0480f5bf6b9c47b49eeccd5da15b60e4: No bootstrap required, opened a new log
I20260811 22:30:21.223461 1531 ts_tablet_manager.cc:1397] T 3104e9581b7544e4880e3bae741abb34 P 0480f5bf6b9c47b49eeccd5da15b60e4: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260811 22:30:21.227533 1525 raft_consensus.cc:348] T 3104e9581b7544e4880e3bae741abb34 P 4ad4667f717e4280bd2a6872f7701f8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad4667f717e4280bd2a6872f7701f8f" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36809 } } peers { permanent_uuid: "a03b794ad15347dba5c0880781152c23" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 38975 } } peers { permanent_uuid: "0480f5bf6b9c47b49eeccd5da15b60e4" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 40067 } }
I20260811 22:30:21.227869 1525 raft_consensus.cc:374] T 3104e9581b7544e4880e3bae741abb34 P 4ad4667f717e4280bd2a6872f7701f8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:30:21.227975 1525 raft_consensus.cc:729] T 3104e9581b7544e4880e3bae741abb34 P 4ad4667f717e4280bd2a6872f7701f8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ad4667f717e4280bd2a6872f7701f8f, State: Initialized, Role: FOLLOWER
I20260811 22:30:21.228556 1525 consensus_queue.cc:260] T 3104e9581b7544e4880e3bae741abb34 P 4ad4667f717e4280bd2a6872f7701f8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad4667f717e4280bd2a6872f7701f8f" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36809 } } peers { permanent_uuid: "a03b794ad15347dba5c0880781152c23" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 38975 } } peers { permanent_uuid: "0480f5bf6b9c47b49eeccd5da15b60e4" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 40067 } }
I20260811 22:30:21.229537 1478 heartbeater.cc:503] Master 127.29.106.62:44179 was elected leader, sending a full tablet report...
I20260811 22:30:21.230192 1525 ts_tablet_manager.cc:1428] T 3104e9581b7544e4880e3bae741abb34 P 4ad4667f717e4280bd2a6872f7701f8f: Time spent starting tablet: real 0.011s user 0.007s sys 0.005s
I20260811 22:30:21.232761 1529 raft_consensus.cc:348] T 3104e9581b7544e4880e3bae741abb34 P a03b794ad15347dba5c0880781152c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad4667f717e4280bd2a6872f7701f8f" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36809 } } peers { permanent_uuid: "a03b794ad15347dba5c0880781152c23" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 38975 } } peers { permanent_uuid: "0480f5bf6b9c47b49eeccd5da15b60e4" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 40067 } }
I20260811 22:30:21.233192 1529 raft_consensus.cc:374] T 3104e9581b7544e4880e3bae741abb34 P a03b794ad15347dba5c0880781152c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:30:21.233300 1529 raft_consensus.cc:729] T 3104e9581b7544e4880e3bae741abb34 P a03b794ad15347dba5c0880781152c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a03b794ad15347dba5c0880781152c23, State: Initialized, Role: FOLLOWER
I20260811 22:30:21.233340 1531 raft_consensus.cc:348] T 3104e9581b7544e4880e3bae741abb34 P 0480f5bf6b9c47b49eeccd5da15b60e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad4667f717e4280bd2a6872f7701f8f" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36809 } } peers { permanent_uuid: "a03b794ad15347dba5c0880781152c23" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 38975 } } peers { permanent_uuid: "0480f5bf6b9c47b49eeccd5da15b60e4" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 40067 } }
I20260811 22:30:21.233749 1531 raft_consensus.cc:374] T 3104e9581b7544e4880e3bae741abb34 P 0480f5bf6b9c47b49eeccd5da15b60e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:30:21.233842 1531 raft_consensus.cc:729] T 3104e9581b7544e4880e3bae741abb34 P 0480f5bf6b9c47b49eeccd5da15b60e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0480f5bf6b9c47b49eeccd5da15b60e4, State: Initialized, Role: FOLLOWER
I20260811 22:30:21.234018 1529 consensus_queue.cc:260] T 3104e9581b7544e4880e3bae741abb34 P a03b794ad15347dba5c0880781152c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad4667f717e4280bd2a6872f7701f8f" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36809 } } peers { permanent_uuid: "a03b794ad15347dba5c0880781152c23" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 38975 } } peers { permanent_uuid: "0480f5bf6b9c47b49eeccd5da15b60e4" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 40067 } }
I20260811 22:30:21.234519 1531 consensus_queue.cc:260] T 3104e9581b7544e4880e3bae741abb34 P 0480f5bf6b9c47b49eeccd5da15b60e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad4667f717e4280bd2a6872f7701f8f" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36809 } } peers { permanent_uuid: "a03b794ad15347dba5c0880781152c23" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 38975 } } peers { permanent_uuid: "0480f5bf6b9c47b49eeccd5da15b60e4" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 40067 } }
I20260811 22:30:21.241220 954 heartbeater.cc:503] Master 127.29.106.62:44179 was elected leader, sending a full tablet report...
I20260811 22:30:21.242244 1529 ts_tablet_manager.cc:1428] T 3104e9581b7544e4880e3bae741abb34 P a03b794ad15347dba5c0880781152c23: Time spent starting tablet: real 0.020s user 0.019s sys 0.001s
I20260811 22:30:21.242406 1531 ts_tablet_manager.cc:1428] T 3104e9581b7544e4880e3bae741abb34 P 0480f5bf6b9c47b49eeccd5da15b60e4: Time spent starting tablet: real 0.019s user 0.014s sys 0.004s
I20260811 22:30:21.242756 1219 heartbeater.cc:503] Master 127.29.106.62:44179 was elected leader, sending a full tablet report...
W20260811 22:30:21.364583 1481 tablet.cc:2369] T 3104e9581b7544e4880e3bae741abb34 P 4ad4667f717e4280bd2a6872f7701f8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:30:21.385744 1539 raft_consensus.cc:482] T 3104e9581b7544e4880e3bae741abb34 P a03b794ad15347dba5c0880781152c23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:30:21.386063 1539 raft_consensus.cc:504] T 3104e9581b7544e4880e3bae741abb34 P a03b794ad15347dba5c0880781152c23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad4667f717e4280bd2a6872f7701f8f" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36809 } } peers { permanent_uuid: "a03b794ad15347dba5c0880781152c23" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 38975 } } peers { permanent_uuid: "0480f5bf6b9c47b49eeccd5da15b60e4" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 40067 } }
I20260811 22:30:21.387776 1539 leader_election.cc:288] T 3104e9581b7544e4880e3bae741abb34 P a03b794ad15347dba5c0880781152c23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ad4667f717e4280bd2a6872f7701f8f (127.29.106.3:36809), 0480f5bf6b9c47b49eeccd5da15b60e4 (127.29.106.2:40067)
I20260811 22:30:21.396354 1522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:21.388214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:56883 (local address 127.29.106.2:40067)
0811 22:30:21.388463 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:21.388471 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:21.388914 (+ 443us) server_negotiation.cc:407] Connection header received
0811 22:30:21.389176 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:21.389181 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:21.389279 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:21.389372 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:30:21.390147 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:21.391218 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:21.394129 (+ 2911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:21.395347 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:21.395501 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:21.395571 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:21.396009 (+ 438us) server_negotiation.cc:299] Negotiation successful
0811 22:30:21.396128 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260811 22:30:21.396355 1520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:21.388327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:55961 (local address 127.29.106.3:36809)
0811 22:30:21.388512 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:21.388518 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:21.389118 (+ 600us) server_negotiation.cc:407] Connection header received
0811 22:30:21.389202 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:21.389207 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:21.389279 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:21.389372 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:30:21.390147 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:21.391222 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:21.394126 (+ 2904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:21.395358 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:21.395522 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:21.395581 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:21.395991 (+ 410us) server_negotiation.cc:299] Negotiation successful
0811 22:30:21.396114 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260811 22:30:21.396488 1545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:21.388150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.3:36809 (local address 127.29.106.1:55961)
0811 22:30:21.388998 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:21.389013 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:21.389121 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:21.389403 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:21.389408 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:21.389435 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:30:21.389879 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:21.389896 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:21.391374 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:21.391379 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:21.393879 (+ 2500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:21.393895 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:21.395544 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:21.395564 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:21.395777 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:21.395863 (+ 86us) client_negotiation.cc:769] Sending connection context
0811 22:30:21.395955 (+ 92us) client_negotiation.cc:240] Negotiation successful
0811 22:30:21.396087 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"mutex_wait_us":141,"thread_start_us":523,"threads_started":1}
I20260811 22:30:21.396917 1544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:21.388076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.2:40067 (local address 127.29.106.1:56883)
0811 22:30:21.388777 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:21.388798 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:21.388975 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:21.389403 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:21.389408 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:21.389508 (+ 100us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:30:21.389986 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:21.389997 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:21.391374 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:21.391380 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:21.393885 (+ 2505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:21.393895 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:21.395576 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:21.395581 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:21.395782 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:21.395867 (+ 85us) client_negotiation.cc:769] Sending connection context
0811 22:30:21.395957 (+ 90us) client_negotiation.cc:240] Negotiation successful
0811 22:30:21.396087 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"mutex_wait_us":223,"spinlock_wait_cycles":22400,"thread_start_us":395,"threads_started":1}
I20260811 22:30:21.397480 1396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3104e9581b7544e4880e3bae741abb34" candidate_uuid: "a03b794ad15347dba5c0880781152c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ad4667f717e4280bd2a6872f7701f8f" is_pre_election: true
I20260811 22:30:21.398029 1396 raft_consensus.cc:2454] T 3104e9581b7544e4880e3bae741abb34 P 4ad4667f717e4280bd2a6872f7701f8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a03b794ad15347dba5c0880781152c23 in term 0.
I20260811 22:30:21.398001 1125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3104e9581b7544e4880e3bae741abb34" candidate_uuid: "a03b794ad15347dba5c0880781152c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0480f5bf6b9c47b49eeccd5da15b60e4" is_pre_election: true
I20260811 22:30:21.398502 1125 raft_consensus.cc:2454] T 3104e9581b7544e4880e3bae741abb34 P 0480f5bf6b9c47b49eeccd5da15b60e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a03b794ad15347dba5c0880781152c23 in term 0.
I20260811 22:30:21.398636 744 leader_election.cc:302] T 3104e9581b7544e4880e3bae741abb34 P a03b794ad15347dba5c0880781152c23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ad4667f717e4280bd2a6872f7701f8f, a03b794ad15347dba5c0880781152c23; no voters:
I20260811 22:30:21.399010 1539 raft_consensus.cc:2781] T 3104e9581b7544e4880e3bae741abb34 P a03b794ad15347dba5c0880781152c23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:30:21.399134 1539 raft_consensus.cc:482] T 3104e9581b7544e4880e3bae741abb34 P a03b794ad15347dba5c0880781152c23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:30:21.399233 1539 raft_consensus.cc:3037] T 3104e9581b7544e4880e3bae741abb34 P a03b794ad15347dba5c0880781152c23 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:30:21.401363 1539 raft_consensus.cc:504] T 3104e9581b7544e4880e3bae741abb34 P a03b794ad15347dba5c0880781152c23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad4667f717e4280bd2a6872f7701f8f" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36809 } } peers { permanent_uuid: "a03b794ad15347dba5c0880781152c23" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 38975 } } peers { permanent_uuid: "0480f5bf6b9c47b49eeccd5da15b60e4" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 40067 } }
I20260811 22:30:21.401997 1539 leader_election.cc:288] T 3104e9581b7544e4880e3bae741abb34 P a03b794ad15347dba5c0880781152c23 [CANDIDATE]: Term 1 election: Requested vote from peers 4ad4667f717e4280bd2a6872f7701f8f (127.29.106.3:36809), 0480f5bf6b9c47b49eeccd5da15b60e4 (127.29.106.2:40067)
I20260811 22:30:21.402359 1396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3104e9581b7544e4880e3bae741abb34" candidate_uuid: "a03b794ad15347dba5c0880781152c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ad4667f717e4280bd2a6872f7701f8f"
I20260811 22:30:21.402467 1125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3104e9581b7544e4880e3bae741abb34" candidate_uuid: "a03b794ad15347dba5c0880781152c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0480f5bf6b9c47b49eeccd5da15b60e4"
I20260811 22:30:21.402522 1396 raft_consensus.cc:3037] T 3104e9581b7544e4880e3bae741abb34 P 4ad4667f717e4280bd2a6872f7701f8f [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:30:21.402661 1125 raft_consensus.cc:3037] T 3104e9581b7544e4880e3bae741abb34 P 0480f5bf6b9c47b49eeccd5da15b60e4 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:30:21.404724 1396 raft_consensus.cc:2454] T 3104e9581b7544e4880e3bae741abb34 P 4ad4667f717e4280bd2a6872f7701f8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a03b794ad15347dba5c0880781152c23 in term 1.
I20260811 22:30:21.404940 1125 raft_consensus.cc:2454] T 3104e9581b7544e4880e3bae741abb34 P 0480f5bf6b9c47b49eeccd5da15b60e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a03b794ad15347dba5c0880781152c23 in term 1.
I20260811 22:30:21.405100 744 leader_election.cc:302] T 3104e9581b7544e4880e3bae741abb34 P a03b794ad15347dba5c0880781152c23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ad4667f717e4280bd2a6872f7701f8f, a03b794ad15347dba5c0880781152c23; no voters:
I20260811 22:30:21.405398 1539 raft_consensus.cc:2781] T 3104e9581b7544e4880e3bae741abb34 P a03b794ad15347dba5c0880781152c23 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:30:21.405860 1539 raft_consensus.cc:686] T 3104e9581b7544e4880e3bae741abb34 P a03b794ad15347dba5c0880781152c23 [term 1 LEADER]: Becoming Leader. State: Replica: a03b794ad15347dba5c0880781152c23, State: Running, Role: LEADER
I20260811 22:30:21.406306 1539 consensus_queue.cc:237] T 3104e9581b7544e4880e3bae741abb34 P a03b794ad15347dba5c0880781152c23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad4667f717e4280bd2a6872f7701f8f" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36809 } } peers { permanent_uuid: "a03b794ad15347dba5c0880781152c23" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 38975 } } peers { permanent_uuid: "0480f5bf6b9c47b49eeccd5da15b60e4" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 40067 } }
I20260811 22:30:21.410713 593 catalog_manager.cc:5495] T 3104e9581b7544e4880e3bae741abb34 P a03b794ad15347dba5c0880781152c23 reported cstate change: term changed from 0 to 1, leader changed from <none> to a03b794ad15347dba5c0880781152c23 (127.29.106.1). New cstate: current_term: 1 leader_uuid: "a03b794ad15347dba5c0880781152c23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ad4667f717e4280bd2a6872f7701f8f" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a03b794ad15347dba5c0880781152c23" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 38975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0480f5bf6b9c47b49eeccd5da15b60e4" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 40067 } health_report { overall_health: UNKNOWN } } }
W20260811 22:30:21.420266 1221 tablet.cc:2369] T 3104e9581b7544e4880e3bae741abb34 P 0480f5bf6b9c47b49eeccd5da15b60e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:30:21.480615 1518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:21.475511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45478 (local address 127.29.106.1:38975)
0811 22:30:21.475717 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:21.475721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:21.475740 (+ 19us) server_negotiation.cc:407] Connection header received
0811 22:30:21.475918 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:21.475922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:21.476020 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:21.476148 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:30:21.476446 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:21.477145 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:21.478482 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:21.478949 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:21.479049 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:21.479561 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:30:21.480064 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:30:21.480194 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:21.480296 (+ 102us) server_negotiation.cc:299] Negotiation successful
0811 22:30:21.480331 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260811 22:30:21.486249 957 tablet.cc:2369] T 3104e9581b7544e4880e3bae741abb34 P a03b794ad15347dba5c0880781152c23: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:30:21.488598 1396 raft_consensus.cc:1261] T 3104e9581b7544e4880e3bae741abb34 P 4ad4667f717e4280bd2a6872f7701f8f [term 1 FOLLOWER]: Refusing update from remote peer a03b794ad15347dba5c0880781152c23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:30:21.488601 1125 raft_consensus.cc:1261] T 3104e9581b7544e4880e3bae741abb34 P 0480f5bf6b9c47b49eeccd5da15b60e4 [term 1 FOLLOWER]: Refusing update from remote peer a03b794ad15347dba5c0880781152c23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:30:21.489450 1546 consensus_queue.cc:1035] T 3104e9581b7544e4880e3bae741abb34 P a03b794ad15347dba5c0880781152c23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ad4667f717e4280bd2a6872f7701f8f" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 36809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:30:21.489766 1539 consensus_queue.cc:1035] T 3104e9581b7544e4880e3bae741abb34 P a03b794ad15347dba5c0880781152c23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0480f5bf6b9c47b49eeccd5da15b60e4" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 40067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:30:21.501619 1551 mvcc.cc:205] Tried to move back new op lower bound from 7317452478407729152 to 7317452478083772416. Current Snapshot: MvccSnapshot[applied={T|T < 7317452478407729152}]
I20260811 22:30:21.502477 1554 mvcc.cc:205] Tried to move back new op lower bound from 7317452478407729152 to 7317452478083772416. Current Snapshot: MvccSnapshot[applied={T|T < 7317452478407729152}]
I20260811 22:30:21.518947 594 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:30:21.519188 594 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:30:21.522719 594 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d8226669e18f4bca920666d4c0c60a69: Sending DeleteTablet for 3 replicas of tablet 3104e9581b7544e4880e3bae741abb34
I20260811 22:30:21.523773 1354 tablet_service.cc:1514] Processing DeleteTablet for tablet 3104e9581b7544e4880e3bae741abb34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:30:21 UTC) from {username='slave'} at 127.0.0.1:53384
I20260811 22:30:21.523902 837 tablet_service.cc:1514] Processing DeleteTablet for tablet 3104e9581b7544e4880e3bae741abb34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:30:21 UTC) from {username='slave'} at 127.0.0.1:45462
I20260811 22:30:21.525027 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 715
I20260811 22:30:21.525118 1096 tablet_service.cc:1514] Processing DeleteTablet for tablet 3104e9581b7544e4880e3bae741abb34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:30:21 UTC) from {username='slave'} at 127.0.0.1:48324
I20260811 22:30:21.531767 1575 tablet_replica.cc:330] T 3104e9581b7544e4880e3bae741abb34 P 0480f5bf6b9c47b49eeccd5da15b60e4: stopping tablet replica
I20260811 22:30:21.532217 1575 raft_consensus.cc:2229] T 3104e9581b7544e4880e3bae741abb34 P 0480f5bf6b9c47b49eeccd5da15b60e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:30:21.532706 1575 raft_consensus.cc:2258] T 3104e9581b7544e4880e3bae741abb34 P 0480f5bf6b9c47b49eeccd5da15b60e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:30:21.536846 1575 ts_tablet_manager.cc:1905] T 3104e9581b7544e4880e3bae741abb34 P 0480f5bf6b9c47b49eeccd5da15b60e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:30:21.537863 1574 tablet_replica.cc:330] T 3104e9581b7544e4880e3bae741abb34 P a03b794ad15347dba5c0880781152c23: stopping tablet replica
I20260811 22:30:21.538313 1574 raft_consensus.cc:2229] T 3104e9581b7544e4880e3bae741abb34 P a03b794ad15347dba5c0880781152c23 [term 1 LEADER]: Raft consensus shutting down.
I20260811 22:30:21.539255 1573 tablet_replica.cc:330] T 3104e9581b7544e4880e3bae741abb34 P 4ad4667f717e4280bd2a6872f7701f8f: stopping tablet replica
I20260811 22:30:21.539666 1573 raft_consensus.cc:2229] T 3104e9581b7544e4880e3bae741abb34 P 4ad4667f717e4280bd2a6872f7701f8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:30:21.540208 1573 raft_consensus.cc:2258] T 3104e9581b7544e4880e3bae741abb34 P 4ad4667f717e4280bd2a6872f7701f8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:30:21.540594 1574 raft_consensus.cc:2258] T 3104e9581b7544e4880e3bae741abb34 P a03b794ad15347dba5c0880781152c23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:30:21.542196 1573 ts_tablet_manager.cc:1905] T 3104e9581b7544e4880e3bae741abb34 P 4ad4667f717e4280bd2a6872f7701f8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:30:21.544950 1575 ts_tablet_manager.cc:1918] T 3104e9581b7544e4880e3bae741abb34 P 0480f5bf6b9c47b49eeccd5da15b60e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:30:21.545112 1575 log.cc:1196] T 3104e9581b7544e4880e3bae741abb34 P 0480f5bf6b9c47b49eeccd5da15b60e4: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-1/wal/wals/3104e9581b7544e4880e3bae741abb34
I20260811 22:30:21.545688 1575 ts_tablet_manager.cc:1939] T 3104e9581b7544e4880e3bae741abb34 P 0480f5bf6b9c47b49eeccd5da15b60e4: Deleting consensus metadata
I20260811 22:30:21.545771 1574 ts_tablet_manager.cc:1905] T 3104e9581b7544e4880e3bae741abb34 P a03b794ad15347dba5c0880781152c23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:30:21.547631 572 catalog_manager.cc:4841] TS 0480f5bf6b9c47b49eeccd5da15b60e4 (127.29.106.2:40067): tablet 3104e9581b7544e4880e3bae741abb34 (table test-table [id=a277099c39fb4d1788c3d1a848ba2665]) successfully deleted
I20260811 22:30:21.548959 1573 ts_tablet_manager.cc:1918] T 3104e9581b7544e4880e3bae741abb34 P 4ad4667f717e4280bd2a6872f7701f8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:30:21.549122 1573 log.cc:1196] T 3104e9581b7544e4880e3bae741abb34 P 4ad4667f717e4280bd2a6872f7701f8f: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-2/wal/wals/3104e9581b7544e4880e3bae741abb34
I20260811 22:30:21.549660 1573 ts_tablet_manager.cc:1939] T 3104e9581b7544e4880e3bae741abb34 P 4ad4667f717e4280bd2a6872f7701f8f: Deleting consensus metadata
I20260811 22:30:21.551343 571 catalog_manager.cc:4841] TS 4ad4667f717e4280bd2a6872f7701f8f (127.29.106.3:36809): tablet 3104e9581b7544e4880e3bae741abb34 (table test-table [id=a277099c39fb4d1788c3d1a848ba2665]) successfully deleted
I20260811 22:30:21.551678 1518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:21.526092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45484 (local address 127.29.106.1:38975)
0811 22:30:21.526270 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:21.526276 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:21.538954 (+ 12678us) server_negotiation.cc:407] Connection header received
0811 22:30:21.539684 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:21.539689 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:21.539790 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:21.539867 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:21.541029 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:21.542274 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:21.547628 (+ 5354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:21.548240 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:21.548318 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:21.550024 (+ 1706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:21.550035 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:21.550050 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:30:21.550770 (+ 720us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:21.550825 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:30:21.550852 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:21.550954 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:21.551171 (+ 217us) server_negotiation.cc:299] Negotiation successful
0811 22:30:21.551302 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260811 22:30:21.552917 945 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:30:21.646777 1574 ts_tablet_manager.cc:1918] T 3104e9581b7544e4880e3bae741abb34 P a03b794ad15347dba5c0880781152c23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:30:21.647047 1574 log.cc:1196] T 3104e9581b7544e4880e3bae741abb34 P a03b794ad15347dba5c0880781152c23: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786487105757681-30120-0/minicluster-data/ts-0/wal/wals/3104e9581b7544e4880e3bae741abb34
I20260811 22:30:21.647449 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 715
W20260811 22:30:21.672857 567 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.106.1:38975 (error 108)
I20260811 22:30:21.672930 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 965
I20260811 22:30:21.674749 1516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:21.674285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:38975 (local address 127.0.0.1:45492)
0811 22:30:21.674469 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:21.674590 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.1:38975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260811 22:30:21.675019 567 catalog_manager.cc:4584] TS a03b794ad15347dba5c0880781152c23 (127.29.106.1:38975): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3104e9581b7544e4880e3bae741abb34: Network error: Client connection negotiation failed: client connection to 127.29.106.1:38975: connect: Connection refused (error 111)
I20260811 22:30:21.680981 1522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:21.673950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48330 (local address 127.29.106.2:40067)
0811 22:30:21.674166 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:21.674171 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:21.674189 (+ 18us) server_negotiation.cc:407] Connection header received
0811 22:30:21.674250 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:21.674255 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:21.674375 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:21.674484 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:21.674944 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:21.676206 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:21.677460 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:21.678140 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:21.678249 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:21.679214 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:21.679225 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:21.679242 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:30:21.680211 (+ 969us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:21.680287 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:30:21.680313 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:21.680416 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:21.680523 (+ 107us) server_negotiation.cc:299] Negotiation successful
0811 22:30:21.680628 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260811 22:30:21.681763 1212 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:30:21.728363 1516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:21.727942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:38975 (local address 127.0.0.1:45506)
0811 22:30:21.728137 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:21.728221 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.1:38975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260811 22:30:21.729507 1516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:21.729115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:38975 (local address 127.0.0.1:45514)
0811 22:30:21.729289 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:21.729378 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.1:38975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260811 22:30:21.768817 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 965
I20260811 22:30:21.790023 1516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:21.789571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:38975 (local address 127.0.0.1:45526)
0811 22:30:21.789769 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:21.789858 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.1:38975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260811 22:30:21.791030 1516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:21.790711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:38975 (local address 127.0.0.1:45536)
0811 22:30:21.790831 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:21.790904 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.1:38975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260811 22:30:21.792580 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 1224
I20260811 22:30:21.798616 1520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:21.793263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53390 (local address 127.29.106.3:36809)
0811 22:30:21.793437 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:21.793442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:21.793460 (+ 18us) server_negotiation.cc:407] Connection header received
0811 22:30:21.793556 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:21.793560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:21.793658 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:21.793738 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:21.794066 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:21.794759 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:21.796043 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:21.796446 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:21.796543 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:21.797455 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:21.797465 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:21.797481 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:30:21.798052 (+ 571us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:21.798100 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:30:21.798121 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:21.798214 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:21.798296 (+ 82us) server_negotiation.cc:299] Negotiation successful
0811 22:30:21.798382 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260811 22:30:21.799232 1470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:30:21.871696 1516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:21.871051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:38975 (local address 127.0.0.1:45542)
0811 22:30:21.871351 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:21.871467 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.1:38975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260811 22:30:21.872833 1516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:21.872358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:38975 (local address 127.0.0.1:45548)
0811 22:30:21.872596 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:21.872691 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.1:38975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260811 22:30:21.885159 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 1224
I20260811 22:30:21.908387 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 536
I20260811 22:30:21.909626 653 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:30:22.023720 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 536
2026-08-11T22:30:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2843 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:30:22 dist-test-slave-gjrz krb5kdc[1589](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:30:22 dist-test-slave-gjrz krb5kdc[1589](info): set up 2 sockets
Aug 11 22:30:22 dist-test-slave-gjrz krb5kdc[1589](info): commencing operation
krb5kdc: starting...
W20260811 22:30:23.286309 30120 mini_kdc.cc:121] Time spent starting KDC: real 1.214s user 0.001s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Aug 11 22:30:23 dist-test-slave-gjrz krb5kdc[1589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:30:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:30:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.106.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
I20260811 22:30:23.559360 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:41895
--webserver_interface=127.29.106.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:44611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:41895
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:30:23.885877 1625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:30:23.886170 1625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:30:23.886337 1625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:30:23.893415 1625 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:30:23.893545 1625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:30:23.893620 1625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:30:23.893697 1625 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:30:23.893752 1625 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:30:23.903880 1625 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:44611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:41895
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:41895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.106.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:30:23.905308 1625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:30:23.906695 1625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:30:23.918637 1633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:23.918677 1645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:23.918712 1635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:30:23.919617 1625 server_base.cc:1034] running on GCE node
I20260811 22:30:23.920374 1625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:30:23.921762 1625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:30:23.922971 1625 hybrid_clock.cc:648] HybridClock initialized: now 1786487423922899 us; error 53 us; skew 500 ppm
I20260811 22:30:23.923349 1625 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:30:23 dist-test-slave-gjrz krb5kdc[1589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487423, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:30:23.931000 1625 init.cc:377] Logged in from keytab as kudu/127.29.106.62@KRBTEST.COM (short username kudu)
I20260811 22:30:23.932603 1625 webserver.cc:460] Webserver started at http://127.29.106.62:35457/ using document root <none> and password file <none>
I20260811 22:30:23.933265 1625 fs_manager.cc:359] Metadata directory not provided
I20260811 22:30:23.933369 1625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:30:23.933630 1625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:30:23.936618 1625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "3dde2ca8c30b44cba1673b50ccc98800"
format_stamp: "Formatted at 2026-08-11 22:30:23 on dist-test-slave-gjrz"
server_key: "07acda95d8c21c1e3bd5b5571f467b56"
server_key_iv: "b098e4060c9304045a06b7eb7439f001"
server_key_version: "encryptionkey@0"
I20260811 22:30:23.937315 1625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "3dde2ca8c30b44cba1673b50ccc98800"
format_stamp: "Formatted at 2026-08-11 22:30:23 on dist-test-slave-gjrz"
server_key: "07acda95d8c21c1e3bd5b5571f467b56"
server_key_iv: "b098e4060c9304045a06b7eb7439f001"
server_key_version: "encryptionkey@0"
I20260811 22:30:23.942711 1625 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260811 22:30:23.946410 1659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:23.947814 1625 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:30:23.947989 1625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "3dde2ca8c30b44cba1673b50ccc98800"
format_stamp: "Formatted at 2026-08-11 22:30:23 on dist-test-slave-gjrz"
server_key: "07acda95d8c21c1e3bd5b5571f467b56"
server_key_iv: "b098e4060c9304045a06b7eb7439f001"
server_key_version: "encryptionkey@0"
I20260811 22:30:23.948169 1625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:30:23.966681 1625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:30:23.970803 1625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:30:23.971047 1625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:30:23.990103 1625 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.62:41895
I20260811 22:30:23.990157 1772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.62:41895 every 8 connection(s)
I20260811 22:30:23.991528 1625 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:30:23.992882 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 1625
I20260811 22:30:23.993052 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:30:23.993541 30120 external_mini_cluster.cc:1428] Setting key 2d86f0bff2e8363411ff9f7d356c517c
I20260811 22:30:23.996022 1774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 11 22:30:24 dist-test-slave-gjrz krb5kdc[1589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:30:24.008862 1774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dde2ca8c30b44cba1673b50ccc98800: Bootstrap starting.
I20260811 22:30:24.012384 1774 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3dde2ca8c30b44cba1673b50ccc98800: Neither blocks nor log segments found. Creating new log.
I20260811 22:30:24.013581 1774 log.cc:824] T 00000000000000000000000000000000 P 3dde2ca8c30b44cba1673b50ccc98800: Log is configured to *not* fsync() on all Append() calls
I20260811 22:30:24.016516 1778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:23.995376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36054 (local address 127.29.106.62:41895)
0811 22:30:23.996177 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:23.996189 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:23.996229 (+ 40us) server_negotiation.cc:407] Connection header received
0811 22:30:23.997913 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:23.997934 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:23.998316 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:23.998636 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:24.000421 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:24.001793 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:24.003657 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:24.004331 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:24.004455 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:24.009379 (+ 4924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:24.009386 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:24.009400 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:30:24.009437 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:24.012463 (+ 3026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:24.013266 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:24.013273 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:24.013279 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:24.013460 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:24.013873 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:24.013877 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:24.013879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:24.014544 (+ 665us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:24.014853 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:24.015134 (+ 281us) server_negotiation.cc:299] Negotiation successful
0811 22:30:24.015544 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":352,"threads_started":1}
I20260811 22:30:24.016655 1774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dde2ca8c30b44cba1673b50ccc98800: No bootstrap required, opened a new log
I20260811 22:30:24.024075 1774 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3dde2ca8c30b44cba1673b50ccc98800 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dde2ca8c30b44cba1673b50ccc98800" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 41895 } }
I20260811 22:30:24.024467 1774 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3dde2ca8c30b44cba1673b50ccc98800 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:30:24.024572 1774 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3dde2ca8c30b44cba1673b50ccc98800 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dde2ca8c30b44cba1673b50ccc98800, State: Initialized, Role: FOLLOWER
I20260811 22:30:24.025136 1774 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3dde2ca8c30b44cba1673b50ccc98800 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dde2ca8c30b44cba1673b50ccc98800" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 41895 } }
I20260811 22:30:24.025352 1774 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3dde2ca8c30b44cba1673b50ccc98800 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:30:24.025491 1774 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3dde2ca8c30b44cba1673b50ccc98800 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:30:24.025651 1774 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3dde2ca8c30b44cba1673b50ccc98800 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:30:24.028088 1774 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3dde2ca8c30b44cba1673b50ccc98800 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dde2ca8c30b44cba1673b50ccc98800" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 41895 } }
I20260811 22:30:24.028566 1774 leader_election.cc:302] T 00000000000000000000000000000000 P 3dde2ca8c30b44cba1673b50ccc98800 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3dde2ca8c30b44cba1673b50ccc98800; no voters:
I20260811 22:30:24.029132 1774 leader_election.cc:288] T 00000000000000000000000000000000 P 3dde2ca8c30b44cba1673b50ccc98800 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:30:24.029263 1781 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3dde2ca8c30b44cba1673b50ccc98800 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:30:24.029919 1781 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3dde2ca8c30b44cba1673b50ccc98800 [term 1 LEADER]: Becoming Leader. State: Replica: 3dde2ca8c30b44cba1673b50ccc98800, State: Running, Role: LEADER
I20260811 22:30:24.030438 1781 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3dde2ca8c30b44cba1673b50ccc98800 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dde2ca8c30b44cba1673b50ccc98800" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 41895 } }
I20260811 22:30:24.031123 1774 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3dde2ca8c30b44cba1673b50ccc98800 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:30:24.033330 1783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dde2ca8c30b44cba1673b50ccc98800 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3dde2ca8c30b44cba1673b50ccc98800" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dde2ca8c30b44cba1673b50ccc98800" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 41895 } } }
I20260811 22:30:24.033731 1783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dde2ca8c30b44cba1673b50ccc98800 [sys.catalog]: This master's current role is: LEADER
I20260811 22:30:24.034478 1784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dde2ca8c30b44cba1673b50ccc98800 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3dde2ca8c30b44cba1673b50ccc98800. Latest consensus state: current_term: 1 leader_uuid: "3dde2ca8c30b44cba1673b50ccc98800" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dde2ca8c30b44cba1673b50ccc98800" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 41895 } } }
I20260811 22:30:24.034793 1784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dde2ca8c30b44cba1673b50ccc98800 [sys.catalog]: This master's current role is: LEADER
I20260811 22:30:24.036350 1794 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:30:24.041257 1794 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:30:24.049953 1794 catalog_manager.cc:1347] Generated new cluster ID: 861e3b2f2c8246eeb4a76df8ffa49591
I20260811 22:30:24.050064 1794 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:30:24.065302 1794 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:30:24.067139 1794 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:30:24.094686 1794 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3dde2ca8c30b44cba1673b50ccc98800: Generated new TSK 0
I20260811 22:30:24.095804 1794 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.106.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab.
Entry for principal kudu/127.29.106.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab.
I20260811 22:30:24.163854 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.1:0
--local_ip_for_outbound_sockets=127.29.106.1
--webserver_interface=127.29.106.1
--webserver_port=0
--tserver_master_addrs=127.29.106.62:41895
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab
--principal=kudu/127.29.106.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:44611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:30:24.494839 1813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:30:24.495092 1813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:30:24.495184 1813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:30:24.502045 1813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:30:24.502138 1813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:30:24.502252 1813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.1
I20260811 22:30:24.511281 1813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:44611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.1.keytab
--principal=kudu/127.29.106.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.106.1
--webserver_port=0
--tserver_master_addrs=127.29.106.62:41895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.1
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:30:24.512383 1813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:30:24.513590 1813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:30:24.526467 1827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:24.526549 1822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:24.526521 1823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:30:24.527582 1813 server_base.cc:1034] running on GCE node
I20260811 22:30:24.528122 1813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:30:24.528975 1813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:30:24.530177 1813 hybrid_clock.cc:648] HybridClock initialized: now 1786487424530128 us; error 41 us; skew 500 ppm
I20260811 22:30:24.530470 1813 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:30:24 dist-test-slave-gjrz krb5kdc[1589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487424, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:30:24.533629 1813 init.cc:377] Logged in from keytab as kudu/127.29.106.1@KRBTEST.COM (short username kudu)
I20260811 22:30:24.535061 1813 webserver.cc:460] Webserver started at http://127.29.106.1:42771/ using document root <none> and password file <none>
I20260811 22:30:24.535743 1813 fs_manager.cc:359] Metadata directory not provided
I20260811 22:30:24.535861 1813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:30:24.536161 1813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:30:24.538898 1813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-0/data/instance:
uuid: "94a05a22bd134c29b1bef716a5b72cca"
format_stamp: "Formatted at 2026-08-11 22:30:24 on dist-test-slave-gjrz"
server_key: "b0bd9cbea7f1b8b2019601cac3eea621"
server_key_iv: "cb0e361f0fcbe8ee37c4bb85ee9d307c"
server_key_version: "encryptionkey@0"
I20260811 22:30:24.539649 1813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance:
uuid: "94a05a22bd134c29b1bef716a5b72cca"
format_stamp: "Formatted at 2026-08-11 22:30:24 on dist-test-slave-gjrz"
server_key: "b0bd9cbea7f1b8b2019601cac3eea621"
server_key_iv: "cb0e361f0fcbe8ee37c4bb85ee9d307c"
server_key_version: "encryptionkey@0"
I20260811 22:30:24.544567 1813 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260811 22:30:24.547945 1843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:24.549182 1813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 22:30:24.549335 1813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-0/wal
uuid: "94a05a22bd134c29b1bef716a5b72cca"
format_stamp: "Formatted at 2026-08-11 22:30:24 on dist-test-slave-gjrz"
server_key: "b0bd9cbea7f1b8b2019601cac3eea621"
server_key_iv: "cb0e361f0fcbe8ee37c4bb85ee9d307c"
server_key_version: "encryptionkey@0"
I20260811 22:30:24.549533 1813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:30:24.581205 1813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:30:24.585325 1813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:30:24.585594 1813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:30:24.586521 1813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:30:24.587816 1813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:30:24.587894 1813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:24.588023 1813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:30:24.588078 1813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:24.618242 1813 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.1:46441
I20260811 22:30:24.618306 2047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.1:46441 every 8 connection(s)
I20260811 22:30:24.619544 1813 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
I20260811 22:30:24.621927 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 1813
I20260811 22:30:24.622121 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance
I20260811 22:30:24.622653 30120 external_mini_cluster.cc:1428] Setting key 9a97b6948ddb92982bbc2be0e9c48c0b
Aug 11 22:30:24 dist-test-slave-gjrz krb5kdc[1589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487424, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.1@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:30:24.639504 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:24.621994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:59439 (local address 127.29.106.62:41895)
0811 22:30:24.622495 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:24.622500 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:24.623681 (+ 1181us) server_negotiation.cc:407] Connection header received
0811 22:30:24.625799 (+ 2118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:24.625805 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:24.625882 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:24.625984 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:24.627478 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:24.628203 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:24.629733 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:24.630135 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:24.630250 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:24.633944 (+ 3694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:24.633949 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:24.633952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:30:24.633984 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:24.636438 (+ 2454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:24.637240 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:24.637243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:24.637246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:24.637363 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:24.637778 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:24.637781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:24.637783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:24.638031 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:24.638189 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:24.638840 (+ 651us) server_negotiation.cc:299] Negotiation successful
0811 22:30:24.639109 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":250,"threads_started":1}
I20260811 22:30:24.640168 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:24.622664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:41895 (local address 127.29.106.1:59439)
0811 22:30:24.623500 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:24.623571 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:24.625550 (+ 1979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:24.626222 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:24.626230 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:24.626752 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:30:24.627322 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:24.627344 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:24.628358 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:24.628362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:24.629598 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:24.629611 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:24.630364 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:24.630371 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:24.630597 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:24.631221 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:30:24.631247 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:30:24.633742 (+ 2495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:30:24.636582 (+ 2840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:30:24.636591 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:30:24.636607 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:30:24.637139 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:30:24.637452 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:30:24.637456 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:30:24.637459 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:30:24.637692 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:30:24.638150 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:30:24.638158 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:30:24.638467 (+ 309us) client_negotiation.cc:769] Sending connection context
0811 22:30:24.638693 (+ 226us) client_negotiation.cc:240] Negotiation successful
0811 22:30:24.639071 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":317,"threads_started":1}
I20260811 22:30:24.641713 2049 heartbeater.cc:347] Connected to a master server at 127.29.106.62:41895
I20260811 22:30:24.642057 2049 heartbeater.cc:464] Registering TS with master...
I20260811 22:30:24.642946 2049 heartbeater.cc:511] Master 127.29.106.62:41895 requested a full tablet report, sending...
I20260811 22:30:24.645143 1689 ts_manager.cc:194] Registered new tserver with Master: 94a05a22bd134c29b1bef716a5b72cca (127.29.106.1:46441)
I20260811 22:30:24.646564 1689 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.1@KRBTEST.COM'} at 127.29.106.1:59439
WARNING: no policy specified for kudu/127.29.106.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab.
Entry for principal kudu/127.29.106.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab.
I20260811 22:30:24.691188 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.2:0
--local_ip_for_outbound_sockets=127.29.106.2
--webserver_interface=127.29.106.2
--webserver_port=0
--tserver_master_addrs=127.29.106.62:41895
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab
--principal=kudu/127.29.106.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:44611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:30:25.025978 2063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:30:25.026237 2063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:30:25.026355 2063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:30:25.033444 2063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:30:25.033545 2063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:30:25.033718 2063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.2
I20260811 22:30:25.043128 2063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:44611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.2.keytab
--principal=kudu/127.29.106.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.106.2
--webserver_port=0
--tserver_master_addrs=127.29.106.62:41895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.2
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:30:25.044314 2063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:30:25.045552 2063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:30:25.057971 2072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:30:25.058568 2063 server_base.cc:1034] running on GCE node
W20260811 22:30:25.059502 2076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:25.059538 2073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:30:25.060209 2063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:30:25.061053 2063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:30:25.062252 2063 hybrid_clock.cc:648] HybridClock initialized: now 1786487425062204 us; error 36 us; skew 500 ppm
I20260811 22:30:25.062534 2063 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:30:25 dist-test-slave-gjrz krb5kdc[1589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487425, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:30:25.065693 2063 init.cc:377] Logged in from keytab as kudu/127.29.106.2@KRBTEST.COM (short username kudu)
I20260811 22:30:25.067040 2063 webserver.cc:460] Webserver started at http://127.29.106.2:33729/ using document root <none> and password file <none>
I20260811 22:30:25.067725 2063 fs_manager.cc:359] Metadata directory not provided
I20260811 22:30:25.067845 2063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:30:25.068149 2063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:30:25.070950 2063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-1/data/instance:
uuid: "f5f716b5155a4d59954772bec78493ff"
format_stamp: "Formatted at 2026-08-11 22:30:25 on dist-test-slave-gjrz"
server_key: "3cb661a5f1067525ef0fe3df56d4e30a"
server_key_iv: "c0b33c89a4e5edf114f21f0f147e4300"
server_key_version: "encryptionkey@0"
I20260811 22:30:25.071700 2063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance:
uuid: "f5f716b5155a4d59954772bec78493ff"
format_stamp: "Formatted at 2026-08-11 22:30:25 on dist-test-slave-gjrz"
server_key: "3cb661a5f1067525ef0fe3df56d4e30a"
server_key_iv: "c0b33c89a4e5edf114f21f0f147e4300"
server_key_version: "encryptionkey@0"
I20260811 22:30:25.076506 2063 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260811 22:30:25.079803 2089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:25.080992 2063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 22:30:25.081153 2063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-1/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-1/wal
uuid: "f5f716b5155a4d59954772bec78493ff"
format_stamp: "Formatted at 2026-08-11 22:30:25 on dist-test-slave-gjrz"
server_key: "3cb661a5f1067525ef0fe3df56d4e30a"
server_key_iv: "c0b33c89a4e5edf114f21f0f147e4300"
server_key_version: "encryptionkey@0"
I20260811 22:30:25.081326 2063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:30:25.100813 2063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:30:25.104351 2063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:30:25.104619 2063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:30:25.105494 2063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:30:25.106701 2063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:30:25.106777 2063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:25.106897 2063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:30:25.106957 2063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:25.135892 2063 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.2:41859
I20260811 22:30:25.135939 2308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.2:41859 every 8 connection(s)
I20260811 22:30:25.137106 2063 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
I20260811 22:30:25.138062 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 2063
I20260811 22:30:25.138247 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance
I20260811 22:30:25.138779 30120 external_mini_cluster.cc:1428] Setting key 169c4b8fdb2c5f0fc525c9f57cfec920
Aug 11 22:30:25 dist-test-slave-gjrz krb5kdc[1589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487425, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.2@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:30:25.157303 2316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:25.140040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:58523 (local address 127.29.106.62:41895)
0811 22:30:25.140427 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:25.140431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:25.141928 (+ 1497us) server_negotiation.cc:407] Connection header received
0811 22:30:25.143721 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:25.143726 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:25.143806 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:25.143899 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:25.145355 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:25.146089 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:25.147620 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:25.148045 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:25.148156 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:25.151360 (+ 3204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:25.151366 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:25.151370 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:30:25.151402 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:25.154239 (+ 2837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:25.154918 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:25.154924 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:25.154928 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:25.155090 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:25.155522 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:25.155526 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:25.155529 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:25.155862 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:25.156001 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:25.156687 (+ 686us) server_negotiation.cc:299] Negotiation successful
0811 22:30:25.156893 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":198,"threads_started":1}
I20260811 22:30:25.158004 2317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:25.140449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:41895 (local address 127.29.106.2:58523)
0811 22:30:25.141763 (+ 1314us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:25.141804 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:25.143473 (+ 1669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:25.144143 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:25.144150 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:25.144654 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:30:25.145191 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:25.145209 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:25.146235 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:25.146239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:25.147485 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:25.147498 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:25.148261 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:25.148268 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:25.148479 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:25.149083 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:30:25.149110 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:30:25.151127 (+ 2017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:30:25.154371 (+ 3244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:30:25.154378 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:30:25.154389 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:30:25.154802 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:30:25.155193 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:30:25.155197 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:30:25.155199 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:30:25.155415 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:30:25.155974 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:30:25.155979 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:30:25.156335 (+ 356us) client_negotiation.cc:769] Sending connection context
0811 22:30:25.156551 (+ 216us) client_negotiation.cc:240] Negotiation successful
0811 22:30:25.156911 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1100,"thread_start_us":325,"threads_started":1}
I20260811 22:30:25.159400 2311 heartbeater.cc:347] Connected to a master server at 127.29.106.62:41895
I20260811 22:30:25.159686 2311 heartbeater.cc:464] Registering TS with master...
I20260811 22:30:25.160358 2311 heartbeater.cc:511] Master 127.29.106.62:41895 requested a full tablet report, sending...
I20260811 22:30:25.161744 1689 ts_manager.cc:194] Registered new tserver with Master: f5f716b5155a4d59954772bec78493ff (127.29.106.2:41859)
I20260811 22:30:25.162534 1689 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.2@KRBTEST.COM'} at 127.29.106.2:58523
WARNING: no policy specified for kudu/127.29.106.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab.
Entry for principal kudu/127.29.106.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab.
I20260811 22:30:25.203848 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.3:0
--local_ip_for_outbound_sockets=127.29.106.3
--webserver_interface=127.29.106.3
--webserver_port=0
--tserver_master_addrs=127.29.106.62:41895
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab
--principal=kudu/127.29.106.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:44611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:30:25.527386 2325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:30:25.527653 2325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:30:25.527803 2325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:30:25.534963 2325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:30:25.535070 2325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:30:25.535220 2325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.3
I20260811 22:30:25.544484 2325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:44611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.3.keytab
--principal=kudu/127.29.106.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.106.3
--webserver_port=0
--tserver_master_addrs=127.29.106.62:41895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.3
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:30:25.545639 2325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:30:25.546851 2325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:30:25.559639 2335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:25.559693 2339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:25.559724 2336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:30:25.560424 2325 server_base.cc:1034] running on GCE node
I20260811 22:30:25.560990 2325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:30:25.561849 2325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:30:25.563054 2325 hybrid_clock.cc:648] HybridClock initialized: now 1786487425563006 us; error 38 us; skew 500 ppm
I20260811 22:30:25.563396 2325 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:30:25 dist-test-slave-gjrz krb5kdc[1589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487425, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:30:25.566282 2325 init.cc:377] Logged in from keytab as kudu/127.29.106.3@KRBTEST.COM (short username kudu)
I20260811 22:30:25.567662 2325 webserver.cc:460] Webserver started at http://127.29.106.3:44519/ using document root <none> and password file <none>
I20260811 22:30:25.568286 2325 fs_manager.cc:359] Metadata directory not provided
I20260811 22:30:25.568405 2325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:30:25.568693 2325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:30:25.571544 2325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-2/data/instance:
uuid: "cfa4c519ddcb47fb86d2731693df52be"
format_stamp: "Formatted at 2026-08-11 22:30:25 on dist-test-slave-gjrz"
server_key: "f7696ad1ac4c4e85b006a43ef2728921"
server_key_iv: "f0861753d7abbecb6ff6cf59700bc6be"
server_key_version: "encryptionkey@0"
I20260811 22:30:25.572271 2325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance:
uuid: "cfa4c519ddcb47fb86d2731693df52be"
format_stamp: "Formatted at 2026-08-11 22:30:25 on dist-test-slave-gjrz"
server_key: "f7696ad1ac4c4e85b006a43ef2728921"
server_key_iv: "f0861753d7abbecb6ff6cf59700bc6be"
server_key_version: "encryptionkey@0"
I20260811 22:30:25.577001 2325 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 22:30:25.580204 2352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:25.581393 2325 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:30:25.581542 2325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-2/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-2/wal
uuid: "cfa4c519ddcb47fb86d2731693df52be"
format_stamp: "Formatted at 2026-08-11 22:30:25 on dist-test-slave-gjrz"
server_key: "f7696ad1ac4c4e85b006a43ef2728921"
server_key_iv: "f0861753d7abbecb6ff6cf59700bc6be"
server_key_version: "encryptionkey@0"
I20260811 22:30:25.581718 2325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:30:25.600834 2325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:30:25.604486 2325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:30:25.604741 2325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:30:25.605624 2325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:30:25.606914 2325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:30:25.606993 2325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:25.607121 2325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:30:25.607175 2325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:25.637671 2325 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.3:40889
I20260811 22:30:25.637732 2570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.3:40889 every 8 connection(s)
I20260811 22:30:25.638888 2325 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
I20260811 22:30:25.648306 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 2325
I20260811 22:30:25.648463 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance
I20260811 22:30:25.648867 30120 external_mini_cluster.cc:1428] Setting key dd4340fb866664af9a2c8e14d858a30b
I20260811 22:30:25.649893 2049 heartbeater.cc:503] Master 127.29.106.62:41895 was elected leader, sending a full tablet report...
Aug 11 22:30:25 dist-test-slave-gjrz krb5kdc[1589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487425, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.3@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:30:25.658577 2316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:25.641225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:34731 (local address 127.29.106.62:41895)
0811 22:30:25.641436 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:25.641443 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:25.642594 (+ 1151us) server_negotiation.cc:407] Connection header received
0811 22:30:25.644409 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:25.644414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:25.644496 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:25.644588 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:25.645962 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:25.646717 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:25.648298 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:25.648765 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:25.648878 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:25.653160 (+ 4282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:25.653166 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:25.653171 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:30:25.653203 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:25.655708 (+ 2505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:25.656381 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:25.656385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:25.656387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:25.656502 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:25.656916 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:25.656921 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:25.656925 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:25.657225 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:25.657363 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:25.658030 (+ 667us) server_negotiation.cc:299] Negotiation successful
0811 22:30:25.658261 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260811 22:30:25.659494 2575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:25.641693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:41895 (local address 127.29.106.3:34731)
0811 22:30:25.642432 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:25.642472 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:25.644164 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:25.644779 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:25.644786 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:25.645282 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:30:25.645826 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:25.645843 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:25.646848 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:25.646851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:25.648152 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:25.648165 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:25.649006 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:25.649014 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:25.649230 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:25.650136 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:30:25.650172 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:30:25.652947 (+ 2775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:30:25.655834 (+ 2887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:30:25.655842 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:30:25.655857 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:30:25.656261 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:30:25.656604 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:30:25.656609 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:30:25.656613 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:30:25.656801 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:30:25.657354 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:30:25.657362 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:30:25.657672 (+ 310us) client_negotiation.cc:769] Sending connection context
0811 22:30:25.657913 (+ 241us) client_negotiation.cc:240] Negotiation successful
0811 22:30:25.658385 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":292,"threads_started":1}
I20260811 22:30:25.660667 2572 heartbeater.cc:347] Connected to a master server at 127.29.106.62:41895
I20260811 22:30:25.660904 2572 heartbeater.cc:464] Registering TS with master...
I20260811 22:30:25.661527 2572 heartbeater.cc:511] Master 127.29.106.62:41895 requested a full tablet report, sending...
I20260811 22:30:25.662927 1688 ts_manager.cc:194] Registered new tserver with Master: cfa4c519ddcb47fb86d2731693df52be (127.29.106.3:40889)
I20260811 22:30:25.663944 1688 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.106.3@KRBTEST.COM'} at 127.29.106.3:34731
I20260811 22:30:25.664158 30120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 22:30:25.680667 2316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:25.668740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36068 (local address 127.29.106.62:41895)
0811 22:30:25.668902 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:25.668908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:25.669276 (+ 368us) server_negotiation.cc:407] Connection header received
0811 22:30:25.669430 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:25.669435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:25.669505 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:25.669586 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:25.670996 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:25.671764 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:25.673017 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:25.673409 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:25.673515 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:25.675815 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:25.675820 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:25.675823 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:30:25.675851 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:25.678195 (+ 2344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:25.678783 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:25.678787 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:25.678789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:25.678909 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:25.679251 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:25.679256 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:25.679260 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:25.679585 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:25.679836 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:25.680144 (+ 308us) server_negotiation.cc:299] Negotiation successful
0811 22:30:25.680344 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 22:30:25.684343 1688 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36068:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:30:25.691114 1688 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:30:25.713380 2593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:25.702625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53860 (local address 127.29.106.3:40889)
0811 22:30:25.703859 (+ 1234us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:25.703868 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:25.704391 (+ 523us) server_negotiation.cc:407] Connection header received
0811 22:30:25.704823 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:25.704831 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:25.705098 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:25.705266 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:30:25.706314 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:25.707838 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:25.710519 (+ 2681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:25.711919 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:25.712015 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:25.712243 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:25.712869 (+ 626us) server_negotiation.cc:299] Negotiation successful
0811 22:30:25.713048 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1068,"thread_start_us":916,"threads_started":1}
I20260811 22:30:25.714250 2592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:25.702375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.3:40889 (local address 127.0.0.1:53860)
0811 22:30:25.704238 (+ 1863us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:25.704281 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:25.704547 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:25.705594 (+ 1047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:25.705600 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:25.705629 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:30:25.706091 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:25.706105 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:25.708049 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:25.708054 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:25.710335 (+ 2281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:25.710347 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:25.712437 (+ 2090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:25.712443 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:25.712617 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:25.712669 (+ 52us) client_negotiation.cc:769] Sending connection context
0811 22:30:25.713728 (+ 1059us) client_negotiation.cc:240] Negotiation successful
0811 22:30:25.713908 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1600,"thread_start_us":1472,"threads_started":1}
I20260811 22:30:25.716290 2594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:25.703156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:46441 (local address 127.0.0.1:59454)
0811 22:30:25.704870 (+ 1714us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:25.704886 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:25.704990 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:25.706960 (+ 1970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:25.706966 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:25.706990 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:30:25.707423 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:25.707436 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:25.710256 (+ 2820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:25.710261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:25.712358 (+ 2097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:25.712375 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:25.715471 (+ 3096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:25.715478 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:25.715668 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:25.715700 (+ 32us) client_negotiation.cc:769] Sending connection context
0811 22:30:25.715773 (+ 73us) client_negotiation.cc:240] Negotiation successful
0811 22:30:25.715910 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1568,"mutex_wait_us":1,"thread_start_us":1428,"threads_started":1}
I20260811 22:30:25.716461 2598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:25.703722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59454 (local address 127.29.106.1:46441)
0811 22:30:25.704444 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:25.704451 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:25.705119 (+ 668us) server_negotiation.cc:407] Connection header received
0811 22:30:25.706288 (+ 1169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:25.706295 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:25.706528 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:25.706727 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:30:25.707607 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:25.710100 (+ 2493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:25.712551 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:25.714961 (+ 2410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:25.715094 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:25.715322 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:25.715989 (+ 667us) server_negotiation.cc:299] Negotiation successful
0811 22:30:25.716161 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":392,"threads_started":1}
I20260811 22:30:25.719350 2447 tablet_service.cc:1467] Processing CreateTablet for tablet b54d8e89a950428db3e3a86cb195dc0b (DEFAULT_TABLE table=test-table [id=89030d6690d64d53badedc2f610a2992]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:30:25.719573 1944 tablet_service.cc:1467] Processing CreateTablet for tablet b54d8e89a950428db3e3a86cb195dc0b (DEFAULT_TABLE table=test-table [id=89030d6690d64d53badedc2f610a2992]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:30:25.720722 1944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b54d8e89a950428db3e3a86cb195dc0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:30:25.720827 2447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b54d8e89a950428db3e3a86cb195dc0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:30:25.722368 2592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:25.704664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.2:41859 (local address 127.0.0.1:34266)
0811 22:30:25.714538 (+ 9874us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:25.714556 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:25.714644 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:25.715708 (+ 1064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:25.715713 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:25.715741 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:30:25.715916 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:25.715927 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:25.718156 (+ 2229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:25.718162 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:25.719929 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:25.719944 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:25.721663 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:25.721669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:25.721843 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:25.721891 (+ 48us) client_negotiation.cc:769] Sending connection context
0811 22:30:25.721991 (+ 100us) client_negotiation.cc:240] Negotiation successful
0811 22:30:25.722125 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9775,"thread_start_us":10549,"threads_started":1}
I20260811 22:30:25.722687 2601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:25.705083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34266 (local address 127.29.106.2:41859)
0811 22:30:25.715138 (+ 10055us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:25.715146 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:25.715171 (+ 25us) server_negotiation.cc:407] Connection header received
0811 22:30:25.715284 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:25.715294 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:25.715513 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:25.715701 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:30:25.716091 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:25.717978 (+ 1887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:25.720119 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:25.721482 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:25.721625 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:25.721818 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:25.722050 (+ 232us) server_negotiation.cc:299] Negotiation successful
0811 22:30:25.722212 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9879,"thread_start_us":9756,"threads_started":1}
I20260811 22:30:25.726182 2193 tablet_service.cc:1467] Processing CreateTablet for tablet b54d8e89a950428db3e3a86cb195dc0b (DEFAULT_TABLE table=test-table [id=89030d6690d64d53badedc2f610a2992]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:30:25.727584 2193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b54d8e89a950428db3e3a86cb195dc0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:30:25.732237 2602 tablet_bootstrap.cc:492] T b54d8e89a950428db3e3a86cb195dc0b P 94a05a22bd134c29b1bef716a5b72cca: Bootstrap starting.
I20260811 22:30:25.735430 2602 tablet_bootstrap.cc:654] T b54d8e89a950428db3e3a86cb195dc0b P 94a05a22bd134c29b1bef716a5b72cca: Neither blocks nor log segments found. Creating new log.
I20260811 22:30:25.736402 2602 log.cc:824] T b54d8e89a950428db3e3a86cb195dc0b P 94a05a22bd134c29b1bef716a5b72cca: Log is configured to *not* fsync() on all Append() calls
I20260811 22:30:25.740883 2605 tablet_bootstrap.cc:492] T b54d8e89a950428db3e3a86cb195dc0b P cfa4c519ddcb47fb86d2731693df52be: Bootstrap starting.
I20260811 22:30:25.742291 2607 tablet_bootstrap.cc:492] T b54d8e89a950428db3e3a86cb195dc0b P f5f716b5155a4d59954772bec78493ff: Bootstrap starting.
I20260811 22:30:25.743793 2602 tablet_bootstrap.cc:492] T b54d8e89a950428db3e3a86cb195dc0b P 94a05a22bd134c29b1bef716a5b72cca: No bootstrap required, opened a new log
I20260811 22:30:25.744033 2602 ts_tablet_manager.cc:1397] T b54d8e89a950428db3e3a86cb195dc0b P 94a05a22bd134c29b1bef716a5b72cca: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.005s
I20260811 22:30:25.744483 2605 tablet_bootstrap.cc:654] T b54d8e89a950428db3e3a86cb195dc0b P cfa4c519ddcb47fb86d2731693df52be: Neither blocks nor log segments found. Creating new log.
I20260811 22:30:25.745698 2607 tablet_bootstrap.cc:654] T b54d8e89a950428db3e3a86cb195dc0b P f5f716b5155a4d59954772bec78493ff: Neither blocks nor log segments found. Creating new log.
I20260811 22:30:25.745714 2605 log.cc:824] T b54d8e89a950428db3e3a86cb195dc0b P cfa4c519ddcb47fb86d2731693df52be: Log is configured to *not* fsync() on all Append() calls
I20260811 22:30:25.746790 2607 log.cc:824] T b54d8e89a950428db3e3a86cb195dc0b P f5f716b5155a4d59954772bec78493ff: Log is configured to *not* fsync() on all Append() calls
I20260811 22:30:25.748595 2605 tablet_bootstrap.cc:492] T b54d8e89a950428db3e3a86cb195dc0b P cfa4c519ddcb47fb86d2731693df52be: No bootstrap required, opened a new log
I20260811 22:30:25.748867 2605 ts_tablet_manager.cc:1397] T b54d8e89a950428db3e3a86cb195dc0b P cfa4c519ddcb47fb86d2731693df52be: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260811 22:30:25.749670 2607 tablet_bootstrap.cc:492] T b54d8e89a950428db3e3a86cb195dc0b P f5f716b5155a4d59954772bec78493ff: No bootstrap required, opened a new log
I20260811 22:30:25.749970 2607 ts_tablet_manager.cc:1397] T b54d8e89a950428db3e3a86cb195dc0b P f5f716b5155a4d59954772bec78493ff: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260811 22:30:25.751986 2602 raft_consensus.cc:348] T b54d8e89a950428db3e3a86cb195dc0b P 94a05a22bd134c29b1bef716a5b72cca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa4c519ddcb47fb86d2731693df52be" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 40889 } } peers { permanent_uuid: "f5f716b5155a4d59954772bec78493ff" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 41859 } } peers { permanent_uuid: "94a05a22bd134c29b1bef716a5b72cca" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 46441 } }
I20260811 22:30:25.752350 2602 raft_consensus.cc:374] T b54d8e89a950428db3e3a86cb195dc0b P 94a05a22bd134c29b1bef716a5b72cca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:30:25.752457 2602 raft_consensus.cc:729] T b54d8e89a950428db3e3a86cb195dc0b P 94a05a22bd134c29b1bef716a5b72cca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94a05a22bd134c29b1bef716a5b72cca, State: Initialized, Role: FOLLOWER
I20260811 22:30:25.753103 2602 consensus_queue.cc:260] T b54d8e89a950428db3e3a86cb195dc0b P 94a05a22bd134c29b1bef716a5b72cca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa4c519ddcb47fb86d2731693df52be" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 40889 } } peers { permanent_uuid: "f5f716b5155a4d59954772bec78493ff" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 41859 } } peers { permanent_uuid: "94a05a22bd134c29b1bef716a5b72cca" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 46441 } }
I20260811 22:30:25.756599 2602 ts_tablet_manager.cc:1428] T b54d8e89a950428db3e3a86cb195dc0b P 94a05a22bd134c29b1bef716a5b72cca: Time spent starting tablet: real 0.012s user 0.010s sys 0.002s
I20260811 22:30:25.759167 2605 raft_consensus.cc:348] T b54d8e89a950428db3e3a86cb195dc0b P cfa4c519ddcb47fb86d2731693df52be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa4c519ddcb47fb86d2731693df52be" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 40889 } } peers { permanent_uuid: "f5f716b5155a4d59954772bec78493ff" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 41859 } } peers { permanent_uuid: "94a05a22bd134c29b1bef716a5b72cca" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 46441 } }
I20260811 22:30:25.759605 2605 raft_consensus.cc:374] T b54d8e89a950428db3e3a86cb195dc0b P cfa4c519ddcb47fb86d2731693df52be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:30:25.759717 2605 raft_consensus.cc:729] T b54d8e89a950428db3e3a86cb195dc0b P cfa4c519ddcb47fb86d2731693df52be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfa4c519ddcb47fb86d2731693df52be, State: Initialized, Role: FOLLOWER
I20260811 22:30:25.760010 2607 raft_consensus.cc:348] T b54d8e89a950428db3e3a86cb195dc0b P f5f716b5155a4d59954772bec78493ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa4c519ddcb47fb86d2731693df52be" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 40889 } } peers { permanent_uuid: "f5f716b5155a4d59954772bec78493ff" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 41859 } } peers { permanent_uuid: "94a05a22bd134c29b1bef716a5b72cca" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 46441 } }
I20260811 22:30:25.760416 2607 raft_consensus.cc:374] T b54d8e89a950428db3e3a86cb195dc0b P f5f716b5155a4d59954772bec78493ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:30:25.760314 2605 consensus_queue.cc:260] T b54d8e89a950428db3e3a86cb195dc0b P cfa4c519ddcb47fb86d2731693df52be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa4c519ddcb47fb86d2731693df52be" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 40889 } } peers { permanent_uuid: "f5f716b5155a4d59954772bec78493ff" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 41859 } } peers { permanent_uuid: "94a05a22bd134c29b1bef716a5b72cca" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 46441 } }
I20260811 22:30:25.760512 2607 raft_consensus.cc:729] T b54d8e89a950428db3e3a86cb195dc0b P f5f716b5155a4d59954772bec78493ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5f716b5155a4d59954772bec78493ff, State: Initialized, Role: FOLLOWER
I20260811 22:30:25.761097 2607 consensus_queue.cc:260] T b54d8e89a950428db3e3a86cb195dc0b P f5f716b5155a4d59954772bec78493ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa4c519ddcb47fb86d2731693df52be" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 40889 } } peers { permanent_uuid: "f5f716b5155a4d59954772bec78493ff" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 41859 } } peers { permanent_uuid: "94a05a22bd134c29b1bef716a5b72cca" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 46441 } }
I20260811 22:30:25.761952 2572 heartbeater.cc:503] Master 127.29.106.62:41895 was elected leader, sending a full tablet report...
I20260811 22:30:25.762177 2605 ts_tablet_manager.cc:1428] T b54d8e89a950428db3e3a86cb195dc0b P cfa4c519ddcb47fb86d2731693df52be: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260811 22:30:25.762740 2607 ts_tablet_manager.cc:1428] T b54d8e89a950428db3e3a86cb195dc0b P f5f716b5155a4d59954772bec78493ff: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260811 22:30:25.762811 2311 heartbeater.cc:503] Master 127.29.106.62:41895 was elected leader, sending a full tablet report...
W20260811 22:30:25.871845 2052 tablet.cc:2369] T b54d8e89a950428db3e3a86cb195dc0b P 94a05a22bd134c29b1bef716a5b72cca: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 22:30:25.888986 2315 tablet.cc:2369] T b54d8e89a950428db3e3a86cb195dc0b P f5f716b5155a4d59954772bec78493ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 22:30:25.890378 2574 tablet.cc:2369] T b54d8e89a950428db3e3a86cb195dc0b P cfa4c519ddcb47fb86d2731693df52be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:30:26.061167 2614 raft_consensus.cc:482] T b54d8e89a950428db3e3a86cb195dc0b P cfa4c519ddcb47fb86d2731693df52be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:30:26.061532 2614 raft_consensus.cc:504] T b54d8e89a950428db3e3a86cb195dc0b P cfa4c519ddcb47fb86d2731693df52be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa4c519ddcb47fb86d2731693df52be" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 40889 } } peers { permanent_uuid: "f5f716b5155a4d59954772bec78493ff" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 41859 } } peers { permanent_uuid: "94a05a22bd134c29b1bef716a5b72cca" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 46441 } }
I20260811 22:30:26.063380 2614 leader_election.cc:288] T b54d8e89a950428db3e3a86cb195dc0b P cfa4c519ddcb47fb86d2731693df52be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5f716b5155a4d59954772bec78493ff (127.29.106.2:41859), 94a05a22bd134c29b1bef716a5b72cca (127.29.106.1:46441)
I20260811 22:30:26.069796 2601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:26.063792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:34023 (local address 127.29.106.2:41859)
0811 22:30:26.064033 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:26.064039 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:26.064281 (+ 242us) server_negotiation.cc:407] Connection header received
0811 22:30:26.064423 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:26.064428 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:26.064509 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:26.064611 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:30:26.064995 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:26.065989 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:26.067936 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:26.068961 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:26.069073 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:26.069124 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:26.069453 (+ 329us) server_negotiation.cc:299] Negotiation successful
0811 22:30:26.069537 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260811 22:30:26.069840 2575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:26.063775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.2:41859 (local address 127.29.106.3:34023)
0811 22:30:26.064111 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:26.064132 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:26.064279 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:26.064636 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:26.064640 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:26.064664 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:30:26.064850 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:26.064861 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:26.066115 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:26.066121 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:26.067771 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:26.067784 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:26.069106 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:26.069112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:26.069305 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:26.069341 (+ 36us) client_negotiation.cc:769] Sending connection context
0811 22:30:26.069438 (+ 97us) client_negotiation.cc:240] Negotiation successful
0811 22:30:26.069573 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":27}
I20260811 22:30:26.070420 2598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:26.063904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:45133 (local address 127.29.106.1:46441)
0811 22:30:26.064224 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:26.064230 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:26.064736 (+ 506us) server_negotiation.cc:407] Connection header received
0811 22:30:26.064894 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:26.064899 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:26.064977 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:26.065062 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:30:26.065649 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:26.066555 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:26.068569 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:26.069478 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:26.069614 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:26.069680 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:26.070061 (+ 381us) server_negotiation.cc:299] Negotiation successful
0811 22:30:26.070190 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176}
I20260811 22:30:26.070786 2224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b54d8e89a950428db3e3a86cb195dc0b" candidate_uuid: "cfa4c519ddcb47fb86d2731693df52be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5f716b5155a4d59954772bec78493ff" is_pre_election: true
I20260811 22:30:26.070984 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:26.063816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:46441 (local address 127.29.106.3:45133)
0811 22:30:26.064635 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:26.064653 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:26.064776 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:26.065084 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:26.065088 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:26.065114 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:30:26.065506 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:26.065519 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:26.066693 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:26.066697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:26.068422 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:26.068437 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:26.069671 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:26.069678 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:26.069865 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:26.069899 (+ 34us) client_negotiation.cc:769] Sending connection context
0811 22:30:26.070580 (+ 681us) client_negotiation.cc:240] Negotiation successful
0811 22:30:26.070691 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"mutex_wait_us":19,"thread_start_us":429,"threads_started":1}
I20260811 22:30:26.071337 2224 raft_consensus.cc:2454] T b54d8e89a950428db3e3a86cb195dc0b P f5f716b5155a4d59954772bec78493ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfa4c519ddcb47fb86d2731693df52be in term 0.
I20260811 22:30:26.072057 2358 leader_election.cc:302] T b54d8e89a950428db3e3a86cb195dc0b P cfa4c519ddcb47fb86d2731693df52be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cfa4c519ddcb47fb86d2731693df52be, f5f716b5155a4d59954772bec78493ff; no voters:
I20260811 22:30:26.072041 1972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b54d8e89a950428db3e3a86cb195dc0b" candidate_uuid: "cfa4c519ddcb47fb86d2731693df52be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94a05a22bd134c29b1bef716a5b72cca" is_pre_election: true
I20260811 22:30:26.072424 2614 raft_consensus.cc:2781] T b54d8e89a950428db3e3a86cb195dc0b P cfa4c519ddcb47fb86d2731693df52be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:30:26.072582 2614 raft_consensus.cc:482] T b54d8e89a950428db3e3a86cb195dc0b P cfa4c519ddcb47fb86d2731693df52be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:30:26.072513 1972 raft_consensus.cc:2454] T b54d8e89a950428db3e3a86cb195dc0b P 94a05a22bd134c29b1bef716a5b72cca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfa4c519ddcb47fb86d2731693df52be in term 0.
I20260811 22:30:26.072683 2614 raft_consensus.cc:3037] T b54d8e89a950428db3e3a86cb195dc0b P cfa4c519ddcb47fb86d2731693df52be [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:30:26.074935 2614 raft_consensus.cc:504] T b54d8e89a950428db3e3a86cb195dc0b P cfa4c519ddcb47fb86d2731693df52be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa4c519ddcb47fb86d2731693df52be" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 40889 } } peers { permanent_uuid: "f5f716b5155a4d59954772bec78493ff" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 41859 } } peers { permanent_uuid: "94a05a22bd134c29b1bef716a5b72cca" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 46441 } }
I20260811 22:30:26.075497 2614 leader_election.cc:288] T b54d8e89a950428db3e3a86cb195dc0b P cfa4c519ddcb47fb86d2731693df52be [CANDIDATE]: Term 1 election: Requested vote from peers f5f716b5155a4d59954772bec78493ff (127.29.106.2:41859), 94a05a22bd134c29b1bef716a5b72cca (127.29.106.1:46441)
I20260811 22:30:26.075908 1972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b54d8e89a950428db3e3a86cb195dc0b" candidate_uuid: "cfa4c519ddcb47fb86d2731693df52be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94a05a22bd134c29b1bef716a5b72cca"
I20260811 22:30:26.075888 2224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b54d8e89a950428db3e3a86cb195dc0b" candidate_uuid: "cfa4c519ddcb47fb86d2731693df52be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5f716b5155a4d59954772bec78493ff"
I20260811 22:30:26.076098 1972 raft_consensus.cc:3037] T b54d8e89a950428db3e3a86cb195dc0b P 94a05a22bd134c29b1bef716a5b72cca [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:30:26.076104 2224 raft_consensus.cc:3037] T b54d8e89a950428db3e3a86cb195dc0b P f5f716b5155a4d59954772bec78493ff [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:30:26.079141 2224 raft_consensus.cc:2454] T b54d8e89a950428db3e3a86cb195dc0b P f5f716b5155a4d59954772bec78493ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfa4c519ddcb47fb86d2731693df52be in term 1.
I20260811 22:30:26.079141 1972 raft_consensus.cc:2454] T b54d8e89a950428db3e3a86cb195dc0b P 94a05a22bd134c29b1bef716a5b72cca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfa4c519ddcb47fb86d2731693df52be in term 1.
I20260811 22:30:26.079547 2358 leader_election.cc:302] T b54d8e89a950428db3e3a86cb195dc0b P cfa4c519ddcb47fb86d2731693df52be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cfa4c519ddcb47fb86d2731693df52be, f5f716b5155a4d59954772bec78493ff; no voters:
I20260811 22:30:26.079830 2614 raft_consensus.cc:2781] T b54d8e89a950428db3e3a86cb195dc0b P cfa4c519ddcb47fb86d2731693df52be [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:30:26.080160 2614 raft_consensus.cc:686] T b54d8e89a950428db3e3a86cb195dc0b P cfa4c519ddcb47fb86d2731693df52be [term 1 LEADER]: Becoming Leader. State: Replica: cfa4c519ddcb47fb86d2731693df52be, State: Running, Role: LEADER
I20260811 22:30:26.080577 2614 consensus_queue.cc:237] T b54d8e89a950428db3e3a86cb195dc0b P cfa4c519ddcb47fb86d2731693df52be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa4c519ddcb47fb86d2731693df52be" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 40889 } } peers { permanent_uuid: "f5f716b5155a4d59954772bec78493ff" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 41859 } } peers { permanent_uuid: "94a05a22bd134c29b1bef716a5b72cca" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 46441 } }
I20260811 22:30:26.085134 1688 catalog_manager.cc:5495] T b54d8e89a950428db3e3a86cb195dc0b P cfa4c519ddcb47fb86d2731693df52be reported cstate change: term changed from 0 to 1, leader changed from <none> to cfa4c519ddcb47fb86d2731693df52be (127.29.106.3). New cstate: current_term: 1 leader_uuid: "cfa4c519ddcb47fb86d2731693df52be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfa4c519ddcb47fb86d2731693df52be" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 40889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5f716b5155a4d59954772bec78493ff" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 41859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94a05a22bd134c29b1bef716a5b72cca" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 46441 } health_report { overall_health: UNKNOWN } } }
I20260811 22:30:26.162182 2593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:26.157115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53874 (local address 127.29.106.3:40889)
0811 22:30:26.157283 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:26.157288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:26.157376 (+ 88us) server_negotiation.cc:407] Connection header received
0811 22:30:26.157562 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:26.157566 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:26.157656 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:26.157808 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:30:26.158174 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:26.158961 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:26.160340 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:26.160743 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:26.160837 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:26.161252 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:30:26.161738 (+ 486us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:30:26.161845 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:26.161905 (+ 60us) server_negotiation.cc:299] Negotiation successful
0811 22:30:26.161979 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 22:30:26.170087 2224 raft_consensus.cc:1261] T b54d8e89a950428db3e3a86cb195dc0b P f5f716b5155a4d59954772bec78493ff [term 1 FOLLOWER]: Refusing update from remote peer cfa4c519ddcb47fb86d2731693df52be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:30:26.170085 1972 raft_consensus.cc:1261] T b54d8e89a950428db3e3a86cb195dc0b P 94a05a22bd134c29b1bef716a5b72cca [term 1 FOLLOWER]: Refusing update from remote peer cfa4c519ddcb47fb86d2731693df52be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:30:26.171222 2618 consensus_queue.cc:1035] T b54d8e89a950428db3e3a86cb195dc0b P cfa4c519ddcb47fb86d2731693df52be [LEADER]: Connected to new peer: Peer: permanent_uuid: "94a05a22bd134c29b1bef716a5b72cca" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 46441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:30:26.171654 2614 consensus_queue.cc:1035] T b54d8e89a950428db3e3a86cb195dc0b P cfa4c519ddcb47fb86d2731693df52be [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5f716b5155a4d59954772bec78493ff" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 41859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:30:26.193058 2626 mvcc.cc:205] Tried to move back new op lower bound from 7317452497583030272 to 7317452497229185024. Current Snapshot: MvccSnapshot[applied={T|T < 7317452497583030272}]
I20260811 22:30:26.195065 1688 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:30:26.195307 1688 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:30:26.198475 1688 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3dde2ca8c30b44cba1673b50ccc98800: Sending DeleteTablet for 3 replicas of tablet b54d8e89a950428db3e3a86cb195dc0b
I20260811 22:30:26.199784 1944 tablet_service.cc:1514] Processing DeleteTablet for tablet b54d8e89a950428db3e3a86cb195dc0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:30:26 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:59454
I20260811 22:30:26.199863 2447 tablet_service.cc:1514] Processing DeleteTablet for tablet b54d8e89a950428db3e3a86cb195dc0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:30:26 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:53860
I20260811 22:30:26.199874 2193 tablet_service.cc:1514] Processing DeleteTablet for tablet b54d8e89a950428db3e3a86cb195dc0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:30:26 UTC) from {username='kudu', principal='kudu/127.29.106.62@KRBTEST.COM'} at 127.0.0.1:34266
I20260811 22:30:26.201910 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 1813
I20260811 22:30:26.210538 2643 tablet_replica.cc:330] T b54d8e89a950428db3e3a86cb195dc0b P cfa4c519ddcb47fb86d2731693df52be: stopping tablet replica
I20260811 22:30:26.211072 2643 raft_consensus.cc:2229] T b54d8e89a950428db3e3a86cb195dc0b P cfa4c519ddcb47fb86d2731693df52be [term 1 LEADER]: Raft consensus shutting down.
I20260811 22:30:26.211796 2643 raft_consensus.cc:2258] T b54d8e89a950428db3e3a86cb195dc0b P cfa4c519ddcb47fb86d2731693df52be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:30:26.213955 2643 ts_tablet_manager.cc:1905] T b54d8e89a950428db3e3a86cb195dc0b P cfa4c519ddcb47fb86d2731693df52be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:30:26.214535 2644 tablet_replica.cc:330] T b54d8e89a950428db3e3a86cb195dc0b P 94a05a22bd134c29b1bef716a5b72cca: stopping tablet replica
I20260811 22:30:26.214967 2644 raft_consensus.cc:2229] T b54d8e89a950428db3e3a86cb195dc0b P 94a05a22bd134c29b1bef716a5b72cca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:30:26.219663 2644 raft_consensus.cc:2258] T b54d8e89a950428db3e3a86cb195dc0b P 94a05a22bd134c29b1bef716a5b72cca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:30:26.221789 2644 ts_tablet_manager.cc:1905] T b54d8e89a950428db3e3a86cb195dc0b P 94a05a22bd134c29b1bef716a5b72cca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:30:26.227763 2646 tablet_replica.cc:330] T b54d8e89a950428db3e3a86cb195dc0b P f5f716b5155a4d59954772bec78493ff: stopping tablet replica
I20260811 22:30:26.228196 2646 raft_consensus.cc:2229] T b54d8e89a950428db3e3a86cb195dc0b P f5f716b5155a4d59954772bec78493ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:30:26.228619 2646 raft_consensus.cc:2258] T b54d8e89a950428db3e3a86cb195dc0b P f5f716b5155a4d59954772bec78493ff [term 1 FOLLOWER]: Raft consensus is shut down!
Aug 11 22:30:26 dist-test-slave-gjrz krb5kdc[1589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.1@KRBTEST.COM
I20260811 22:30:26.230646 2646 ts_tablet_manager.cc:1905] T b54d8e89a950428db3e3a86cb195dc0b P f5f716b5155a4d59954772bec78493ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:30:26.232414 2643 ts_tablet_manager.cc:1918] T b54d8e89a950428db3e3a86cb195dc0b P cfa4c519ddcb47fb86d2731693df52be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:30:26.232581 2643 log.cc:1196] T b54d8e89a950428db3e3a86cb195dc0b P cfa4c519ddcb47fb86d2731693df52be: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-2/wal/wals/b54d8e89a950428db3e3a86cb195dc0b
I20260811 22:30:26.233146 2643 ts_tablet_manager.cc:1939] T b54d8e89a950428db3e3a86cb195dc0b P cfa4c519ddcb47fb86d2731693df52be: Deleting consensus metadata
I20260811 22:30:26.234735 1667 catalog_manager.cc:4841] TS cfa4c519ddcb47fb86d2731693df52be (127.29.106.3:40889): tablet b54d8e89a950428db3e3a86cb195dc0b (table test-table [id=89030d6690d64d53badedc2f610a2992]) successfully deleted
I20260811 22:30:26.235852 2644 ts_tablet_manager.cc:1918] T b54d8e89a950428db3e3a86cb195dc0b P 94a05a22bd134c29b1bef716a5b72cca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:30:26.236032 2644 log.cc:1196] T b54d8e89a950428db3e3a86cb195dc0b P 94a05a22bd134c29b1bef716a5b72cca: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-0/wal/wals/b54d8e89a950428db3e3a86cb195dc0b
I20260811 22:30:26.236537 2644 ts_tablet_manager.cc:1939] T b54d8e89a950428db3e3a86cb195dc0b P 94a05a22bd134c29b1bef716a5b72cca: Deleting consensus metadata
I20260811 22:30:26.237457 2598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:26.203225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59462 (local address 127.29.106.1:46441)
0811 22:30:26.203391 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:26.203397 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:26.218577 (+ 15180us) server_negotiation.cc:407] Connection header received
0811 22:30:26.219001 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:26.219007 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:26.219085 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:26.219168 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:26.221011 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:26.222124 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:26.224016 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:26.224641 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:26.224729 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:26.230075 (+ 5346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:26.230090 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:26.230105 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:30:26.230144 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:26.233943 (+ 3799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:26.234809 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:26.234818 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:26.234822 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:26.234991 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:26.235476 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:26.235482 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:26.235485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:26.235927 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:26.236255 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:26.236839 (+ 584us) server_negotiation.cc:299] Negotiation successful
0811 22:30:26.237059 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260811 22:30:26.238194 1662 catalog_manager.cc:4841] TS 94a05a22bd134c29b1bef716a5b72cca (127.29.106.1:46441): tablet b54d8e89a950428db3e3a86cb195dc0b (table test-table [id=89030d6690d64d53badedc2f610a2992]) successfully deleted
I20260811 22:30:26.239033 2042 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:30:26.239060 2646 ts_tablet_manager.cc:1918] T b54d8e89a950428db3e3a86cb195dc0b P f5f716b5155a4d59954772bec78493ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:30:26.239197 2646 log.cc:1196] T b54d8e89a950428db3e3a86cb195dc0b P f5f716b5155a4d59954772bec78493ff: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786487105757681-30120-0/minicluster-data/ts-1/wal/wals/b54d8e89a950428db3e3a86cb195dc0b
I20260811 22:30:26.239800 2646 ts_tablet_manager.cc:1939] T b54d8e89a950428db3e3a86cb195dc0b P f5f716b5155a4d59954772bec78493ff: Deleting consensus metadata
I20260811 22:30:26.241856 1667 catalog_manager.cc:4841] TS f5f716b5155a4d59954772bec78493ff (127.29.106.2:41859): tablet b54d8e89a950428db3e3a86cb195dc0b (table test-table [id=89030d6690d64d53badedc2f610a2992]) successfully deleted
I20260811 22:30:26.356333 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 1813
I20260811 22:30:26.385073 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 2063
Aug 11 22:30:26 dist-test-slave-gjrz krb5kdc[1589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.2@KRBTEST.COM
I20260811 22:30:26.402048 2601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:26.385852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34270 (local address 127.29.106.2:41859)
0811 22:30:26.386014 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:26.386019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:26.386190 (+ 171us) server_negotiation.cc:407] Connection header received
0811 22:30:26.386440 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:26.386446 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:26.386544 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:26.386645 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:26.387968 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:26.389072 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:26.390412 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:26.391051 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:26.391179 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:26.395616 (+ 4437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:26.395636 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:26.395657 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:30:26.395699 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:26.399206 (+ 3507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:26.399993 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:26.400003 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:26.400006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:26.400165 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:26.400508 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:26.400514 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:26.400517 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:26.400993 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:26.401294 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:26.401443 (+ 149us) server_negotiation.cc:299] Negotiation successful
0811 22:30:26.401666 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260811 22:30:26.402701 2293 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:30:26.491171 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 2063
I20260811 22:30:26.516047 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 2325
Aug 11 22:30:26 dist-test-slave-gjrz krb5kdc[1589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.3@KRBTEST.COM
I20260811 22:30:26.533726 2593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:26.516814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53886 (local address 127.29.106.3:40889)
0811 22:30:26.517013 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:26.517019 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:26.517041 (+ 22us) server_negotiation.cc:407] Connection header received
0811 22:30:26.517113 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:26.517118 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:26.517248 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:26.517366 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:26.518787 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:26.520060 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:26.521342 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:26.522052 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:26.522185 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:26.526686 (+ 4501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:26.526701 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:26.526723 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:30:26.526768 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:26.530786 (+ 4018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:26.531442 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:26.531453 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:26.531457 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:26.531678 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:26.532070 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:26.532075 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:26.532079 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:26.532546 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:26.532875 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:26.533043 (+ 168us) server_negotiation.cc:299] Negotiation successful
0811 22:30:26.533289 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260811 22:30:26.534487 2563 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:30:26.654980 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 2325
I20260811 22:30:26.679720 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 1625
I20260811 22:30:26.680881 1761 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:30:26.792225 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 1625
2026-08-11T22:30:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4778 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-11T22:30:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:30:26Z Disabled control of system clock
I20260811 22:30:26.891176 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:40725
--webserver_interface=127.29.106.62
--webserver_port=0
--builtin_ntp_servers=127.29.106.20:45885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:40725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 22:30:27.217204 2664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:30:27.217459 2664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:30:27.217582 2664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:30:27.224606 2664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:30:27.224694 2664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:30:27.224754 2664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:30:27.224835 2664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:30:27.224893 2664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:30:27.233696 2664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:45885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:40725
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:40725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.106.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:30:27.234836 2664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:30:27.236516 2664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:30:27.247489 2676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:27.247460 2681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:27.247547 2683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:30:27.248311 2664 server_base.cc:1034] running on GCE node
I20260811 22:30:27.248996 2664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:30:27.250231 2664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:30:27.251415 2664 hybrid_clock.cc:648] HybridClock initialized: now 1786487427251377 us; error 33 us; skew 500 ppm
I20260811 22:30:27.251698 2664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 22:30:27.253682 2664 webserver.cc:460] Webserver started at http://127.29.106.62:42025/ using document root <none> and password file <none>
I20260811 22:30:27.254258 2664 fs_manager.cc:359] Metadata directory not provided
I20260811 22:30:27.254341 2664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:30:27.254588 2664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:30:27.257262 2664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "eeb5555961194908afafba3f73592c53"
format_stamp: "Formatted at 2026-08-11 22:30:27 on dist-test-slave-gjrz"
server_key: "60410cb0a9411732411695bf1553ca51"
server_key_iv: "b9c16fef7d3c9daf54568c8f3c36d6ff"
server_key_version: "encryptionkey@0"
I20260811 22:30:27.257984 2664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "eeb5555961194908afafba3f73592c53"
format_stamp: "Formatted at 2026-08-11 22:30:27 on dist-test-slave-gjrz"
server_key: "60410cb0a9411732411695bf1553ca51"
server_key_iv: "b9c16fef7d3c9daf54568c8f3c36d6ff"
server_key_version: "encryptionkey@0"
I20260811 22:30:27.262697 2664 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 22:30:27.265853 2694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:27.266979 2664 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 22:30:27.267151 2664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "eeb5555961194908afafba3f73592c53"
format_stamp: "Formatted at 2026-08-11 22:30:27 on dist-test-slave-gjrz"
server_key: "60410cb0a9411732411695bf1553ca51"
server_key_iv: "b9c16fef7d3c9daf54568c8f3c36d6ff"
server_key_version: "encryptionkey@0"
I20260811 22:30:27.267381 2664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:30:27.294350 2664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:30:27.295086 2664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:30:27.295373 2664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:30:27.312914 2664 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.62:40725
I20260811 22:30:27.312961 2803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.62:40725 every 8 connection(s)
I20260811 22:30:27.314150 2664 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:30:27.317884 2806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:30:27.323235 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 2664
I20260811 22:30:27.323423 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:30:27.323853 30120 external_mini_cluster.cc:1428] Setting key 4a6b269a836b3d186b3cbf953f79e07b
I20260811 22:30:27.328513 2806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eeb5555961194908afafba3f73592c53: Bootstrap starting.
I20260811 22:30:27.331935 2806 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eeb5555961194908afafba3f73592c53: Neither blocks nor log segments found. Creating new log.
I20260811 22:30:27.333194 2806 log.cc:824] T 00000000000000000000000000000000 P eeb5555961194908afafba3f73592c53: Log is configured to *not* fsync() on all Append() calls
I20260811 22:30:27.336270 2806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eeb5555961194908afafba3f73592c53: No bootstrap required, opened a new log
I20260811 22:30:27.337244 2810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:27.325705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50270 (local address 127.29.106.62:40725)
0811 22:30:27.326748 (+ 1043us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:27.326758 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:27.326799 (+ 41us) server_negotiation.cc:407] Connection header received
0811 22:30:27.328497 (+ 1698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:27.328522 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:27.328954 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:27.329256 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:27.329944 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:27.331216 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:27.333080 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:27.333889 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:27.334082 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:27.335079 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:27.335087 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:27.335101 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:30:27.335640 (+ 539us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:27.335718 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:30:27.335755 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:27.335843 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:27.336068 (+ 225us) server_negotiation.cc:299] Negotiation successful
0811 22:30:27.336365 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":398,"threads_started":1}
I20260811 22:30:27.343542 2806 raft_consensus.cc:348] T 00000000000000000000000000000000 P eeb5555961194908afafba3f73592c53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeb5555961194908afafba3f73592c53" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 40725 } }
I20260811 22:30:27.343887 2806 raft_consensus.cc:374] T 00000000000000000000000000000000 P eeb5555961194908afafba3f73592c53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:30:27.343982 2806 raft_consensus.cc:729] T 00000000000000000000000000000000 P eeb5555961194908afafba3f73592c53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeb5555961194908afafba3f73592c53, State: Initialized, Role: FOLLOWER
I20260811 22:30:27.344522 2806 consensus_queue.cc:260] T 00000000000000000000000000000000 P eeb5555961194908afafba3f73592c53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeb5555961194908afafba3f73592c53" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 40725 } }
I20260811 22:30:27.344749 2806 raft_consensus.cc:388] T 00000000000000000000000000000000 P eeb5555961194908afafba3f73592c53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:30:27.344868 2806 raft_consensus.cc:482] T 00000000000000000000000000000000 P eeb5555961194908afafba3f73592c53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:30:27.345072 2806 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eeb5555961194908afafba3f73592c53 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:30:27.347144 2806 raft_consensus.cc:504] T 00000000000000000000000000000000 P eeb5555961194908afafba3f73592c53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeb5555961194908afafba3f73592c53" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 40725 } }
I20260811 22:30:27.347622 2806 leader_election.cc:302] T 00000000000000000000000000000000 P eeb5555961194908afafba3f73592c53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eeb5555961194908afafba3f73592c53; no voters:
I20260811 22:30:27.348205 2806 leader_election.cc:288] T 00000000000000000000000000000000 P eeb5555961194908afafba3f73592c53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:30:27.348294 2812 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eeb5555961194908afafba3f73592c53 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:30:27.349390 2812 raft_consensus.cc:686] T 00000000000000000000000000000000 P eeb5555961194908afafba3f73592c53 [term 1 LEADER]: Becoming Leader. State: Replica: eeb5555961194908afafba3f73592c53, State: Running, Role: LEADER
I20260811 22:30:27.349860 2812 consensus_queue.cc:237] T 00000000000000000000000000000000 P eeb5555961194908afafba3f73592c53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeb5555961194908afafba3f73592c53" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 40725 } }
I20260811 22:30:27.350186 2806 sys_catalog.cc:564] T 00000000000000000000000000000000 P eeb5555961194908afafba3f73592c53 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:30:27.352452 2818 sys_catalog.cc:455] T 00000000000000000000000000000000 P eeb5555961194908afafba3f73592c53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eeb5555961194908afafba3f73592c53. Latest consensus state: current_term: 1 leader_uuid: "eeb5555961194908afafba3f73592c53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeb5555961194908afafba3f73592c53" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 40725 } } }
I20260811 22:30:27.352608 2815 sys_catalog.cc:455] T 00000000000000000000000000000000 P eeb5555961194908afafba3f73592c53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eeb5555961194908afafba3f73592c53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eeb5555961194908afafba3f73592c53" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 40725 } } }
I20260811 22:30:27.352900 2818 sys_catalog.cc:458] T 00000000000000000000000000000000 P eeb5555961194908afafba3f73592c53 [sys.catalog]: This master's current role is: LEADER
I20260811 22:30:27.352927 2815 sys_catalog.cc:458] T 00000000000000000000000000000000 P eeb5555961194908afafba3f73592c53 [sys.catalog]: This master's current role is: LEADER
I20260811 22:30:27.354801 2826 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:30:27.359648 2826 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:30:27.368183 2826 catalog_manager.cc:1347] Generated new cluster ID: f44d9ad28052411b9d9a922a0762acda
I20260811 22:30:27.368286 2826 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:30:27.391356 2826 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:30:27.393040 2826 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:30:27.411863 2826 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eeb5555961194908afafba3f73592c53: Generated new TSK 0
I20260811 22:30:27.412837 2826 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:30:27.419642 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.1:0
--local_ip_for_outbound_sockets=127.29.106.1
--webserver_interface=127.29.106.1
--webserver_port=0
--tserver_master_addrs=127.29.106.62:40725
--builtin_ntp_servers=127.29.106.20:45885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 22:30:27.749974 2844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:30:27.750245 2844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:30:27.750335 2844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:30:27.757381 2844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:30:27.757470 2844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:30:27.757577 2844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.1
I20260811 22:30:27.766283 2844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:45885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.106.1
--webserver_port=0
--tserver_master_addrs=127.29.106.62:40725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.1
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:30:27.767370 2844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:30:27.768987 2844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:30:27.781220 2853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:27.781242 2850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:27.781539 2855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:30:27.781852 2844 server_base.cc:1034] running on GCE node
I20260811 22:30:27.782413 2844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:30:27.783231 2844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:30:27.784447 2844 hybrid_clock.cc:648] HybridClock initialized: now 1786487427784405 us; error 36 us; skew 500 ppm
I20260811 22:30:27.784739 2844 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 22:30:27.786785 2844 webserver.cc:460] Webserver started at http://127.29.106.1:46845/ using document root <none> and password file <none>
I20260811 22:30:27.787439 2844 fs_manager.cc:359] Metadata directory not provided
I20260811 22:30:27.787562 2844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:30:27.787845 2844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:30:27.790498 2844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-0/data/instance:
uuid: "85a2a18986c042cc9b68076059cf1c05"
format_stamp: "Formatted at 2026-08-11 22:30:27 on dist-test-slave-gjrz"
server_key: "fc99cce62b26bf9a8dd61f3dcbcc451a"
server_key_iv: "f1581e46d33b624d0cfbdedc7f8b6a34"
server_key_version: "encryptionkey@0"
I20260811 22:30:27.791240 2844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance:
uuid: "85a2a18986c042cc9b68076059cf1c05"
format_stamp: "Formatted at 2026-08-11 22:30:27 on dist-test-slave-gjrz"
server_key: "fc99cce62b26bf9a8dd61f3dcbcc451a"
server_key_iv: "f1581e46d33b624d0cfbdedc7f8b6a34"
server_key_version: "encryptionkey@0"
I20260811 22:30:27.795904 2844 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260811 22:30:27.799073 2866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:27.800269 2844 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260811 22:30:27.800411 2844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-0/wal
uuid: "85a2a18986c042cc9b68076059cf1c05"
format_stamp: "Formatted at 2026-08-11 22:30:27 on dist-test-slave-gjrz"
server_key: "fc99cce62b26bf9a8dd61f3dcbcc451a"
server_key_iv: "f1581e46d33b624d0cfbdedc7f8b6a34"
server_key_version: "encryptionkey@0"
I20260811 22:30:27.800590 2844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:30:27.819715 2844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:30:27.820386 2844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:30:27.820652 2844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:30:27.821509 2844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:30:27.822822 2844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:30:27.822916 2844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:27.823021 2844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:30:27.823099 2844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:27.853061 2844 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.1:41549
I20260811 22:30:27.853152 3083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.1:41549 every 8 connection(s)
I20260811 22:30:27.854194 2844 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-0/data/info.pb
I20260811 22:30:27.858726 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 2844
I20260811 22:30:27.858906 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-0/wal/instance
I20260811 22:30:27.859472 30120 external_mini_cluster.cc:1428] Setting key d6b3e6cc010c95b0a7fc3517e1e66f30
I20260811 22:30:27.862370 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.2:0
--local_ip_for_outbound_sockets=127.29.106.2
--webserver_interface=127.29.106.2
--webserver_port=0
--tserver_master_addrs=127.29.106.62:40725
--builtin_ntp_servers=127.29.106.20:45885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260811 22:30:27.867565 3087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:27.856622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:55669 (local address 127.29.106.62:40725)
0811 22:30:27.857025 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:27.857029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:27.858174 (+ 1145us) server_negotiation.cc:407] Connection header received
0811 22:30:27.860467 (+ 2293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:27.860473 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:27.860564 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:27.860660 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:27.861870 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:27.862825 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:27.864951 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:27.865352 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:27.865443 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:27.866849 (+ 1406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:27.866854 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:27.866857 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:30:27.866874 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:27.866916 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:30:27.866928 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:27.866992 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:27.867208 (+ 216us) server_negotiation.cc:299] Negotiation successful
0811 22:30:27.867310 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":201,"threads_started":1}
I20260811 22:30:27.868736 3090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:27.857158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:40725 (local address 127.29.106.1:55669)
0811 22:30:27.858023 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:27.858058 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:27.860209 (+ 2151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:27.860895 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:27.860904 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:27.861000 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:30:27.861699 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:27.861725 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:27.862983 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:27.862988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:27.864826 (+ 1838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:27.864843 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:27.865568 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:27.865574 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:27.865847 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:27.866531 (+ 684us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 22:30:27.866535 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:30:27.866583 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 22:30:27.866678 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:30:27.867017 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:30:27.867028 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:30:27.867035 (+ 7us) client_negotiation.cc:769] Sending connection context
0811 22:30:27.867201 (+ 166us) client_negotiation.cc:240] Negotiation successful
0811 22:30:27.867582 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":266,"threads_started":1}
I20260811 22:30:27.870079 3084 heartbeater.cc:347] Connected to a master server at 127.29.106.62:40725
I20260811 22:30:27.870400 3084 heartbeater.cc:464] Registering TS with master...
I20260811 22:30:27.871302 3084 heartbeater.cc:511] Master 127.29.106.62:40725 requested a full tablet report, sending...
I20260811 22:30:27.873297 2727 ts_manager.cc:194] Registered new tserver with Master: 85a2a18986c042cc9b68076059cf1c05 (127.29.106.1:41549)
I20260811 22:30:27.874758 2727 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.106.1:55669
W20260811 22:30:28.189231 3091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:30:28.189481 3091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:30:28.189608 3091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:30:28.196636 3091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:30:28.196725 3091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:30:28.196858 3091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.2
I20260811 22:30:28.205641 3091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:45885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.106.2
--webserver_port=0
--tserver_master_addrs=127.29.106.62:40725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.2
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:30:28.206786 3091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:30:28.208473 3091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:30:28.220455 3105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:28.220407 3103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:28.220551 3101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:30:28.221463 3091 server_base.cc:1034] running on GCE node
I20260811 22:30:28.222078 3091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:30:28.222929 3091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:30:28.224126 3091 hybrid_clock.cc:648] HybridClock initialized: now 1786487428224083 us; error 33 us; skew 500 ppm
I20260811 22:30:28.224411 3091 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 22:30:28.226447 3091 webserver.cc:460] Webserver started at http://127.29.106.2:44967/ using document root <none> and password file <none>
I20260811 22:30:28.227072 3091 fs_manager.cc:359] Metadata directory not provided
I20260811 22:30:28.227169 3091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:30:28.227486 3091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:30:28.230139 3091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-1/data/instance:
uuid: "d33cf74da520476b999edc281500ca1a"
format_stamp: "Formatted at 2026-08-11 22:30:28 on dist-test-slave-gjrz"
server_key: "587b5591d040524104dd061496c8586e"
server_key_iv: "52deec0589af3692e4546e5cce817494"
server_key_version: "encryptionkey@0"
I20260811 22:30:28.230868 3091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance:
uuid: "d33cf74da520476b999edc281500ca1a"
format_stamp: "Formatted at 2026-08-11 22:30:28 on dist-test-slave-gjrz"
server_key: "587b5591d040524104dd061496c8586e"
server_key_iv: "52deec0589af3692e4546e5cce817494"
server_key_version: "encryptionkey@0"
I20260811 22:30:28.235548 3091 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.001s
I20260811 22:30:28.238686 3120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:28.239909 3091 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:30:28.240069 3091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-1/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-1/wal
uuid: "d33cf74da520476b999edc281500ca1a"
format_stamp: "Formatted at 2026-08-11 22:30:28 on dist-test-slave-gjrz"
server_key: "587b5591d040524104dd061496c8586e"
server_key_iv: "52deec0589af3692e4546e5cce817494"
server_key_version: "encryptionkey@0"
I20260811 22:30:28.240255 3091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:30:28.252017 3091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:30:28.252672 3091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:30:28.252931 3091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:30:28.253755 3091 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:30:28.255084 3091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:30:28.255177 3091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:28.255301 3091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:30:28.255380 3091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260811 22:30:28.285534 3091 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.2:40789
I20260811 22:30:28.285583 3319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.2:40789 every 8 connection(s)
I20260811 22:30:28.286744 3091 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-1/data/info.pb
I20260811 22:30:28.292353 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 3091
I20260811 22:30:28.292515 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-1/wal/instance
I20260811 22:30:28.292936 30120 external_mini_cluster.cc:1428] Setting key 72517fbbfa6a786b2ef72c3ebce27244
I20260811 22:30:28.295689 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.106.3:0
--local_ip_for_outbound_sockets=127.29.106.3
--webserver_interface=127.29.106.3
--webserver_port=0
--tserver_master_addrs=127.29.106.62:40725
--builtin_ntp_servers=127.29.106.20:45885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260811 22:30:28.298784 3087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:28.289124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:37817 (local address 127.29.106.62:40725)
0811 22:30:28.289287 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:28.289292 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:28.290602 (+ 1310us) server_negotiation.cc:407] Connection header received
0811 22:30:28.292382 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:28.292388 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:28.292482 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:28.292584 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:28.293712 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:28.294580 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:28.296258 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:28.296670 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:28.296787 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:28.298017 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:28.298024 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:28.298029 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:30:28.298050 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:28.298081 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:30:28.298092 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:28.298167 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:28.298371 (+ 204us) server_negotiation.cc:299] Negotiation successful
0811 22:30:28.298481 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260811 22:30:28.299867 3326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:28.289589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:40725 (local address 127.29.106.2:37817)
0811 22:30:28.290444 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:28.290485 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:28.292106 (+ 1621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:28.292815 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:28.292823 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:28.292904 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:30:28.293517 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:28.293543 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:28.294762 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:28.294767 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:28.296120 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:28.296133 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:28.296880 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:28.296888 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:28.297106 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:28.297730 (+ 624us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 22:30:28.297733 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:30:28.297770 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 22:30:28.297841 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:30:28.298175 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:30:28.298187 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:30:28.298196 (+ 9us) client_negotiation.cc:769] Sending connection context
0811 22:30:28.298346 (+ 150us) client_negotiation.cc:240] Negotiation successful
0811 22:30:28.298702 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":262,"threads_started":1}
I20260811 22:30:28.301115 3322 heartbeater.cc:347] Connected to a master server at 127.29.106.62:40725
I20260811 22:30:28.301411 3322 heartbeater.cc:464] Registering TS with master...
I20260811 22:30:28.302239 3322 heartbeater.cc:511] Master 127.29.106.62:40725 requested a full tablet report, sending...
I20260811 22:30:28.303639 2727 ts_manager.cc:194] Registered new tserver with Master: d33cf74da520476b999edc281500ca1a (127.29.106.2:40789)
I20260811 22:30:28.304489 2727 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.106.2:37817
W20260811 22:30:28.622861 3327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:30:28.623116 3327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:30:28.623209 3327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:30:28.630192 3327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:30:28.630280 3327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:30:28.630388 3327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.106.3
I20260811 22:30:28.638981 3327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:45885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.106.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.106.3
--webserver_port=0
--tserver_master_addrs=127.29.106.62:40725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.106.3
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:30:28.640067 3327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:30:28.641711 3327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:30:28.654276 3334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:28.654299 3335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:28.654371 3337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:30:28.655167 3327 server_base.cc:1034] running on GCE node
I20260811 22:30:28.655817 3327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:30:28.656726 3327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:30:28.657915 3327 hybrid_clock.cc:648] HybridClock initialized: now 1786487428657863 us; error 45 us; skew 500 ppm
I20260811 22:30:28.658201 3327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260811 22:30:28.660215 3327 webserver.cc:460] Webserver started at http://127.29.106.3:41095/ using document root <none> and password file <none>
I20260811 22:30:28.660825 3327 fs_manager.cc:359] Metadata directory not provided
I20260811 22:30:28.660917 3327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:30:28.661201 3327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:30:28.663956 3327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-2/data/instance:
uuid: "b26c21784af645328c1d38a11a7bcbf6"
format_stamp: "Formatted at 2026-08-11 22:30:28 on dist-test-slave-gjrz"
server_key: "a500dc6a4affe88f776385d032e3bf8f"
server_key_iv: "3d98ea1ec54248326c5aad061c0d1d4b"
server_key_version: "encryptionkey@0"
I20260811 22:30:28.664701 3327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance:
uuid: "b26c21784af645328c1d38a11a7bcbf6"
format_stamp: "Formatted at 2026-08-11 22:30:28 on dist-test-slave-gjrz"
server_key: "a500dc6a4affe88f776385d032e3bf8f"
server_key_iv: "3d98ea1ec54248326c5aad061c0d1d4b"
server_key_version: "encryptionkey@0"
I20260811 22:30:28.669360 3327 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 22:30:28.672609 3354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:28.673784 3327 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:30:28.673930 3327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-2/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-2/wal
uuid: "b26c21784af645328c1d38a11a7bcbf6"
format_stamp: "Formatted at 2026-08-11 22:30:28 on dist-test-slave-gjrz"
server_key: "a500dc6a4affe88f776385d032e3bf8f"
server_key_iv: "3d98ea1ec54248326c5aad061c0d1d4b"
server_key_version: "encryptionkey@0"
I20260811 22:30:28.674124 3327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:30:28.688164 3327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:30:28.688936 3327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:30:28.689215 3327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:30:28.690193 3327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:30:28.691692 3327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:30:28.691778 3327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:28.691906 3327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:30:28.691978 3327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:28.722121 3327 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.3:40711
I20260811 22:30:28.722169 3568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.3:40711 every 8 connection(s)
I20260811 22:30:28.723348 3327 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-2/data/info.pb
I20260811 22:30:28.725777 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 3327
I20260811 22:30:28.725940 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-2/wal/instance
I20260811 22:30:28.726420 30120 external_mini_cluster.cc:1428] Setting key 8f2af64060d5c2a55d49affa18c995a5
I20260811 22:30:28.735199 3087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:28.725779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.3:42139 (local address 127.29.106.62:40725)
0811 22:30:28.725989 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:28.725994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:28.727508 (+ 1514us) server_negotiation.cc:407] Connection header received
0811 22:30:28.729219 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:28.729224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:28.729304 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:28.729389 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:28.730414 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:28.731155 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:28.732686 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:28.733099 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:28.733217 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:28.734418 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:28.734423 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:28.734426 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:30:28.734442 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:28.734474 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:30:28.734486 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:28.734569 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:28.734763 (+ 194us) server_negotiation.cc:299] Negotiation successful
0811 22:30:28.734891 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260811 22:30:28.736064 3573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:28.726310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:40725 (local address 127.29.106.3:42139)
0811 22:30:28.727346 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:28.727404 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:28.728968 (+ 1564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:28.729632 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:28.729640 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:28.729721 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:30:28.730258 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:28.730275 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:28.731326 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:28.731332 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:28.732552 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:28.732564 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:28.733306 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:28.733313 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:28.733527 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:28.734132 (+ 605us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 22:30:28.734135 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:30:28.734172 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 22:30:28.734242 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:30:28.734573 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:30:28.734584 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:30:28.734592 (+ 8us) client_negotiation.cc:769] Sending connection context
0811 22:30:28.734739 (+ 147us) client_negotiation.cc:240] Negotiation successful
0811 22:30:28.735098 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":325,"threads_started":1}
I20260811 22:30:28.737174 3569 heartbeater.cc:347] Connected to a master server at 127.29.106.62:40725
I20260811 22:30:28.737396 3569 heartbeater.cc:464] Registering TS with master...
I20260811 22:30:28.738032 3569 heartbeater.cc:511] Master 127.29.106.62:40725 requested a full tablet report, sending...
I20260811 22:30:28.739326 2727 ts_manager.cc:194] Registered new tserver with Master: b26c21784af645328c1d38a11a7bcbf6 (127.29.106.3:40711)
I20260811 22:30:28.740334 2727 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.106.3:42139
I20260811 22:30:28.741101 30120 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 22:30:28.750058 3087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:28.744984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50272 (local address 127.29.106.62:40725)
0811 22:30:28.745119 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:28.745123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:28.745572 (+ 449us) server_negotiation.cc:407] Connection header received
0811 22:30:28.745691 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:28.745694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:28.745758 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:28.745834 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:28.746251 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:28.746883 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:28.748210 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:28.748602 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:28.748687 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:28.749533 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:28.749537 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:28.749541 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:30:28.749556 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:28.749584 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:30:28.749594 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:28.749659 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:28.749737 (+ 78us) server_negotiation.cc:299] Negotiation successful
0811 22:30:28.749803 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260811 22:30:28.753644 2724 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:30:28.760531 2724 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:30:28.781999 3595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:28.772050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.2:40789 (local address 127.0.0.1:57298)
0811 22:30:28.773444 (+ 1394us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:28.773479 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:28.774134 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:28.774858 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:28.774866 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:28.774894 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:30:28.775347 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:28.775360 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:28.776944 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:28.776949 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:28.779402 (+ 2453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:28.779417 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:28.781076 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:28.781083 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:28.781291 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:28.781342 (+ 51us) client_negotiation.cc:769] Sending connection context
0811 22:30:28.781466 (+ 124us) client_negotiation.cc:240] Negotiation successful
0811 22:30:28.781640 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":798,"mutex_wait_us":104,"thread_start_us":680,"threads_started":1}
I20260811 22:30:28.782073 3597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:28.772597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57298 (local address 127.29.106.2:40789)
0811 22:30:28.773264 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:28.773272 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:28.773613 (+ 341us) server_negotiation.cc:407] Connection header received
0811 22:30:28.774382 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:28.774389 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:28.774623 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:28.774850 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:30:28.775602 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:28.776810 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:28.779618 (+ 2808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:28.780883 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:28.781028 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:28.781284 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:28.781525 (+ 241us) server_negotiation.cc:299] Negotiation successful
0811 22:30:28.781706 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":730,"threads_started":1}
I20260811 22:30:28.782925 3596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:28.771881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56316 (local address 127.29.106.3:40711)
0811 22:30:28.772713 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:28.772722 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:28.774483 (+ 1761us) server_negotiation.cc:407] Connection header received
0811 22:30:28.774595 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:28.774601 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:28.774796 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:28.775837 (+ 1041us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:30:28.776102 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:28.777671 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:28.780243 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:28.781315 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:28.782099 (+ 784us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:28.782357 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:28.782432 (+ 75us) server_negotiation.cc:299] Negotiation successful
0811 22:30:28.782584 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":503,"threads_started":1}
I20260811 22:30:28.783321 3600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:28.773695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40836 (local address 127.29.106.1:41549)
0811 22:30:28.774236 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:28.774255 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:28.774738 (+ 483us) server_negotiation.cc:407] Connection header received
0811 22:30:28.774935 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:28.774943 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:28.775167 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:28.775361 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:30:28.776252 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:28.777769 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:28.780499 (+ 2730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:28.781785 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:28.781926 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:28.782491 (+ 565us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:28.782778 (+ 287us) server_negotiation.cc:299] Negotiation successful
0811 22:30:28.782943 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":575,"threads_started":1}
I20260811 22:30:28.783931 3603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:28.773473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:41549 (local address 127.0.0.1:40836)
0811 22:30:28.774631 (+ 1158us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:28.774644 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:28.774756 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:28.775653 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:28.775657 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:28.775687 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:30:28.776059 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:28.776072 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:28.778039 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:28.778045 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:28.780299 (+ 2254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:28.780314 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:28.782402 (+ 2088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:28.782408 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:28.782572 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:28.782595 (+ 23us) client_negotiation.cc:769] Sending connection context
0811 22:30:28.783569 (+ 974us) client_negotiation.cc:240] Negotiation successful
0811 22:30:28.783692 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1057,"mutex_wait_us":1,"thread_start_us":933,"threads_started":1}
I20260811 22:30:28.784340 3594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:28.771787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.3:40711 (local address 127.0.0.1:56316)
0811 22:30:28.773983 (+ 2196us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:28.773998 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:28.774134 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:28.774996 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:28.775001 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:28.775031 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:30:28.775634 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:28.775647 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:28.777836 (+ 2189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:28.777841 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:28.780052 (+ 2211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:28.780067 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:28.781529 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:28.781536 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:28.781684 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:28.781714 (+ 30us) client_negotiation.cc:769] Sending connection context
0811 22:30:28.781775 (+ 61us) client_negotiation.cc:240] Negotiation successful
0811 22:30:28.781890 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":943,"mutex_wait_us":4,"spinlock_wait_cycles":11648,"thread_start_us":962,"threads_started":1}
I20260811 22:30:28.785892 3198 tablet_service.cc:1467] Processing CreateTablet for tablet 763f84e91fa24badbf64ee2f460a91e3 (DEFAULT_TABLE table=test-table [id=59fd38f7773f4e6e8fd4ffca67e05114]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:30:28.786871 2958 tablet_service.cc:1467] Processing CreateTablet for tablet 763f84e91fa24badbf64ee2f460a91e3 (DEFAULT_TABLE table=test-table [id=59fd38f7773f4e6e8fd4ffca67e05114]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:30:28.787493 3198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 763f84e91fa24badbf64ee2f460a91e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:30:28.788141 2958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 763f84e91fa24badbf64ee2f460a91e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:30:28.787830 3436 tablet_service.cc:1467] Processing CreateTablet for tablet 763f84e91fa24badbf64ee2f460a91e3 (DEFAULT_TABLE table=test-table [id=59fd38f7773f4e6e8fd4ffca67e05114]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:30:28.789227 3436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 763f84e91fa24badbf64ee2f460a91e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:30:28.798640 3607 tablet_bootstrap.cc:492] T 763f84e91fa24badbf64ee2f460a91e3 P 85a2a18986c042cc9b68076059cf1c05: Bootstrap starting.
I20260811 22:30:28.802044 3607 tablet_bootstrap.cc:654] T 763f84e91fa24badbf64ee2f460a91e3 P 85a2a18986c042cc9b68076059cf1c05: Neither blocks nor log segments found. Creating new log.
I20260811 22:30:28.802974 3607 log.cc:824] T 763f84e91fa24badbf64ee2f460a91e3 P 85a2a18986c042cc9b68076059cf1c05: Log is configured to *not* fsync() on all Append() calls
I20260811 22:30:28.804287 3612 tablet_bootstrap.cc:492] T 763f84e91fa24badbf64ee2f460a91e3 P b26c21784af645328c1d38a11a7bcbf6: Bootstrap starting.
I20260811 22:30:28.805933 3607 tablet_bootstrap.cc:492] T 763f84e91fa24badbf64ee2f460a91e3 P 85a2a18986c042cc9b68076059cf1c05: No bootstrap required, opened a new log
I20260811 22:30:28.806218 3607 ts_tablet_manager.cc:1397] T 763f84e91fa24badbf64ee2f460a91e3 P 85a2a18986c042cc9b68076059cf1c05: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260811 22:30:28.807664 3609 tablet_bootstrap.cc:492] T 763f84e91fa24badbf64ee2f460a91e3 P d33cf74da520476b999edc281500ca1a: Bootstrap starting.
I20260811 22:30:28.807708 3612 tablet_bootstrap.cc:654] T 763f84e91fa24badbf64ee2f460a91e3 P b26c21784af645328c1d38a11a7bcbf6: Neither blocks nor log segments found. Creating new log.
I20260811 22:30:28.808805 3612 log.cc:824] T 763f84e91fa24badbf64ee2f460a91e3 P b26c21784af645328c1d38a11a7bcbf6: Log is configured to *not* fsync() on all Append() calls
I20260811 22:30:28.810240 3609 tablet_bootstrap.cc:654] T 763f84e91fa24badbf64ee2f460a91e3 P d33cf74da520476b999edc281500ca1a: Neither blocks nor log segments found. Creating new log.
I20260811 22:30:28.812175 3609 log.cc:824] T 763f84e91fa24badbf64ee2f460a91e3 P d33cf74da520476b999edc281500ca1a: Log is configured to *not* fsync() on all Append() calls
I20260811 22:30:28.812860 3612 tablet_bootstrap.cc:492] T 763f84e91fa24badbf64ee2f460a91e3 P b26c21784af645328c1d38a11a7bcbf6: No bootstrap required, opened a new log
I20260811 22:30:28.813120 3612 ts_tablet_manager.cc:1397] T 763f84e91fa24badbf64ee2f460a91e3 P b26c21784af645328c1d38a11a7bcbf6: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260811 22:30:28.814716 3607 raft_consensus.cc:348] T 763f84e91fa24badbf64ee2f460a91e3 P 85a2a18986c042cc9b68076059cf1c05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b26c21784af645328c1d38a11a7bcbf6" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 40711 } } peers { permanent_uuid: "d33cf74da520476b999edc281500ca1a" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 40789 } } peers { permanent_uuid: "85a2a18986c042cc9b68076059cf1c05" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 41549 } }
I20260811 22:30:28.815037 3609 tablet_bootstrap.cc:492] T 763f84e91fa24badbf64ee2f460a91e3 P d33cf74da520476b999edc281500ca1a: No bootstrap required, opened a new log
I20260811 22:30:28.815028 3607 raft_consensus.cc:374] T 763f84e91fa24badbf64ee2f460a91e3 P 85a2a18986c042cc9b68076059cf1c05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:30:28.815126 3607 raft_consensus.cc:729] T 763f84e91fa24badbf64ee2f460a91e3 P 85a2a18986c042cc9b68076059cf1c05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a2a18986c042cc9b68076059cf1c05, State: Initialized, Role: FOLLOWER
I20260811 22:30:28.815322 3609 ts_tablet_manager.cc:1397] T 763f84e91fa24badbf64ee2f460a91e3 P d33cf74da520476b999edc281500ca1a: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.002s
I20260811 22:30:28.815822 3607 consensus_queue.cc:260] T 763f84e91fa24badbf64ee2f460a91e3 P 85a2a18986c042cc9b68076059cf1c05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b26c21784af645328c1d38a11a7bcbf6" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 40711 } } peers { permanent_uuid: "d33cf74da520476b999edc281500ca1a" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 40789 } } peers { permanent_uuid: "85a2a18986c042cc9b68076059cf1c05" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 41549 } }
I20260811 22:30:28.816854 3084 heartbeater.cc:503] Master 127.29.106.62:40725 was elected leader, sending a full tablet report...
I20260811 22:30:28.817466 3607 ts_tablet_manager.cc:1428] T 763f84e91fa24badbf64ee2f460a91e3 P 85a2a18986c042cc9b68076059cf1c05: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260811 22:30:28.823053 3612 raft_consensus.cc:348] T 763f84e91fa24badbf64ee2f460a91e3 P b26c21784af645328c1d38a11a7bcbf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b26c21784af645328c1d38a11a7bcbf6" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 40711 } } peers { permanent_uuid: "d33cf74da520476b999edc281500ca1a" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 40789 } } peers { permanent_uuid: "85a2a18986c042cc9b68076059cf1c05" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 41549 } }
I20260811 22:30:28.823521 3612 raft_consensus.cc:374] T 763f84e91fa24badbf64ee2f460a91e3 P b26c21784af645328c1d38a11a7bcbf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:30:28.823619 3612 raft_consensus.cc:729] T 763f84e91fa24badbf64ee2f460a91e3 P b26c21784af645328c1d38a11a7bcbf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b26c21784af645328c1d38a11a7bcbf6, State: Initialized, Role: FOLLOWER
I20260811 22:30:28.824254 3612 consensus_queue.cc:260] T 763f84e91fa24badbf64ee2f460a91e3 P b26c21784af645328c1d38a11a7bcbf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b26c21784af645328c1d38a11a7bcbf6" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 40711 } } peers { permanent_uuid: "d33cf74da520476b999edc281500ca1a" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 40789 } } peers { permanent_uuid: "85a2a18986c042cc9b68076059cf1c05" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 41549 } }
I20260811 22:30:28.825206 3569 heartbeater.cc:503] Master 127.29.106.62:40725 was elected leader, sending a full tablet report...
I20260811 22:30:28.825212 3609 raft_consensus.cc:348] T 763f84e91fa24badbf64ee2f460a91e3 P d33cf74da520476b999edc281500ca1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b26c21784af645328c1d38a11a7bcbf6" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 40711 } } peers { permanent_uuid: "d33cf74da520476b999edc281500ca1a" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 40789 } } peers { permanent_uuid: "85a2a18986c042cc9b68076059cf1c05" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 41549 } }
I20260811 22:30:28.825635 3609 raft_consensus.cc:374] T 763f84e91fa24badbf64ee2f460a91e3 P d33cf74da520476b999edc281500ca1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:30:28.825793 3609 raft_consensus.cc:729] T 763f84e91fa24badbf64ee2f460a91e3 P d33cf74da520476b999edc281500ca1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d33cf74da520476b999edc281500ca1a, State: Initialized, Role: FOLLOWER
I20260811 22:30:28.826002 3612 ts_tablet_manager.cc:1428] T 763f84e91fa24badbf64ee2f460a91e3 P b26c21784af645328c1d38a11a7bcbf6: Time spent starting tablet: real 0.013s user 0.011s sys 0.002s
I20260811 22:30:28.826381 3609 consensus_queue.cc:260] T 763f84e91fa24badbf64ee2f460a91e3 P d33cf74da520476b999edc281500ca1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b26c21784af645328c1d38a11a7bcbf6" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 40711 } } peers { permanent_uuid: "d33cf74da520476b999edc281500ca1a" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 40789 } } peers { permanent_uuid: "85a2a18986c042cc9b68076059cf1c05" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 41549 } }
I20260811 22:30:28.827436 3322 heartbeater.cc:503] Master 127.29.106.62:40725 was elected leader, sending a full tablet report...
I20260811 22:30:28.827998 3609 ts_tablet_manager.cc:1428] T 763f84e91fa24badbf64ee2f460a91e3 P d33cf74da520476b999edc281500ca1a: Time spent starting tablet: real 0.012s user 0.009s sys 0.003s
W20260811 22:30:28.856127 3085 tablet.cc:2369] T 763f84e91fa24badbf64ee2f460a91e3 P 85a2a18986c042cc9b68076059cf1c05: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:30:28.891090 3623 raft_consensus.cc:482] T 763f84e91fa24badbf64ee2f460a91e3 P d33cf74da520476b999edc281500ca1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:30:28.891448 3623 raft_consensus.cc:504] T 763f84e91fa24badbf64ee2f460a91e3 P d33cf74da520476b999edc281500ca1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b26c21784af645328c1d38a11a7bcbf6" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 40711 } } peers { permanent_uuid: "d33cf74da520476b999edc281500ca1a" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 40789 } } peers { permanent_uuid: "85a2a18986c042cc9b68076059cf1c05" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 41549 } }
I20260811 22:30:28.893182 3623 leader_election.cc:288] T 763f84e91fa24badbf64ee2f460a91e3 P d33cf74da520476b999edc281500ca1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b26c21784af645328c1d38a11a7bcbf6 (127.29.106.3:40711), 85a2a18986c042cc9b68076059cf1c05 (127.29.106.1:41549)
I20260811 22:30:28.900471 3616 raft_consensus.cc:482] T 763f84e91fa24badbf64ee2f460a91e3 P 85a2a18986c042cc9b68076059cf1c05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:30:28.900722 3616 raft_consensus.cc:504] T 763f84e91fa24badbf64ee2f460a91e3 P 85a2a18986c042cc9b68076059cf1c05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b26c21784af645328c1d38a11a7bcbf6" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 40711 } } peers { permanent_uuid: "d33cf74da520476b999edc281500ca1a" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 40789 } } peers { permanent_uuid: "85a2a18986c042cc9b68076059cf1c05" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 41549 } }
I20260811 22:30:28.901197 3600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:28.893655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:40625 (local address 127.29.106.1:41549)
0811 22:30:28.893851 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:28.893857 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:28.894408 (+ 551us) server_negotiation.cc:407] Connection header received
0811 22:30:28.894822 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:28.894826 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:28.894891 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:28.894955 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:30:28.895522 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:28.896502 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:28.898967 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:28.900193 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:28.900341 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:28.900404 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:28.900799 (+ 395us) server_negotiation.cc:299] Negotiation successful
0811 22:30:28.900935 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260811 22:30:28.901276 3625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:28.893494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:41549 (local address 127.29.106.2:40625)
0811 22:30:28.894257 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:28.894270 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:28.894418 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:28.895004 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:28.895008 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:28.895033 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:30:28.895386 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:28.895395 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:28.896624 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:28.896629 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:28.898760 (+ 2131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:28.898776 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:28.900412 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:28.900419 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:28.900621 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:28.900662 (+ 41us) client_negotiation.cc:769] Sending connection context
0811 22:30:28.900760 (+ 98us) client_negotiation.cc:240] Negotiation successful
0811 22:30:28.900936 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"mutex_wait_us":57,"thread_start_us":489,"threads_started":1}
I20260811 22:30:28.901326 3596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:28.893599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.2:59887 (local address 127.29.106.3:40711)
0811 22:30:28.893794 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:28.893801 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:28.894365 (+ 564us) server_negotiation.cc:407] Connection header received
0811 22:30:28.894550 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:28.894554 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:28.894613 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:28.894682 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:30:28.895232 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:28.896309 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:28.898974 (+ 2665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:28.900196 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:28.900870 (+ 674us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:28.900946 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:28.900998 (+ 52us) server_negotiation.cc:299] Negotiation successful
0811 22:30:28.901116 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260811 22:30:28.901783 3624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:28.893393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.3:40711 (local address 127.29.106.2:59887)
0811 22:30:28.894248 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:28.894269 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:28.894424 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:28.894709 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:28.894712 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:28.894732 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:30:28.895093 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:28.895104 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:28.896446 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:28.896450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:28.898767 (+ 2317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:28.898777 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:28.900464 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:28.900469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:28.900652 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:28.900677 (+ 25us) client_negotiation.cc:769] Sending connection context
0811 22:30:28.900749 (+ 72us) client_negotiation.cc:240] Negotiation successful
0811 22:30:28.901518 (+ 769us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"mutex_wait_us":673,"spinlock_wait_cycles":14080,"thread_start_us":466,"threads_started":1}
I20260811 22:30:28.902271 2996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763f84e91fa24badbf64ee2f460a91e3" candidate_uuid: "d33cf74da520476b999edc281500ca1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a2a18986c042cc9b68076059cf1c05" is_pre_election: true
I20260811 22:30:28.902693 3616 leader_election.cc:288] T 763f84e91fa24badbf64ee2f460a91e3 P 85a2a18986c042cc9b68076059cf1c05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b26c21784af645328c1d38a11a7bcbf6 (127.29.106.3:40711), d33cf74da520476b999edc281500ca1a (127.29.106.2:40789)
I20260811 22:30:28.902720 2996 raft_consensus.cc:2454] T 763f84e91fa24badbf64ee2f460a91e3 P 85a2a18986c042cc9b68076059cf1c05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d33cf74da520476b999edc281500ca1a in term 0.
I20260811 22:30:28.902995 3477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763f84e91fa24badbf64ee2f460a91e3" candidate_uuid: "d33cf74da520476b999edc281500ca1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b26c21784af645328c1d38a11a7bcbf6" is_pre_election: true
I20260811 22:30:28.903606 3477 raft_consensus.cc:2454] T 763f84e91fa24badbf64ee2f460a91e3 P b26c21784af645328c1d38a11a7bcbf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d33cf74da520476b999edc281500ca1a in term 0.
I20260811 22:30:28.903785 3126 leader_election.cc:302] T 763f84e91fa24badbf64ee2f460a91e3 P d33cf74da520476b999edc281500ca1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a2a18986c042cc9b68076059cf1c05, d33cf74da520476b999edc281500ca1a; no voters:
I20260811 22:30:28.904312 3623 raft_consensus.cc:2781] T 763f84e91fa24badbf64ee2f460a91e3 P d33cf74da520476b999edc281500ca1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:30:28.904485 3623 raft_consensus.cc:482] T 763f84e91fa24badbf64ee2f460a91e3 P d33cf74da520476b999edc281500ca1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:30:28.904585 3623 raft_consensus.cc:3037] T 763f84e91fa24badbf64ee2f460a91e3 P d33cf74da520476b999edc281500ca1a [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:30:28.907747 3623 raft_consensus.cc:504] T 763f84e91fa24badbf64ee2f460a91e3 P d33cf74da520476b999edc281500ca1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b26c21784af645328c1d38a11a7bcbf6" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 40711 } } peers { permanent_uuid: "d33cf74da520476b999edc281500ca1a" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 40789 } } peers { permanent_uuid: "85a2a18986c042cc9b68076059cf1c05" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 41549 } }
I20260811 22:30:28.908311 3623 leader_election.cc:288] T 763f84e91fa24badbf64ee2f460a91e3 P d33cf74da520476b999edc281500ca1a [CANDIDATE]: Term 1 election: Requested vote from peers b26c21784af645328c1d38a11a7bcbf6 (127.29.106.3:40711), 85a2a18986c042cc9b68076059cf1c05 (127.29.106.1:41549)
I20260811 22:30:28.908704 3477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763f84e91fa24badbf64ee2f460a91e3" candidate_uuid: "d33cf74da520476b999edc281500ca1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b26c21784af645328c1d38a11a7bcbf6"
I20260811 22:30:28.908908 3477 raft_consensus.cc:3037] T 763f84e91fa24badbf64ee2f460a91e3 P b26c21784af645328c1d38a11a7bcbf6 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:30:28.908941 2996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763f84e91fa24badbf64ee2f460a91e3" candidate_uuid: "d33cf74da520476b999edc281500ca1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85a2a18986c042cc9b68076059cf1c05"
I20260811 22:30:28.909152 2996 raft_consensus.cc:3037] T 763f84e91fa24badbf64ee2f460a91e3 P 85a2a18986c042cc9b68076059cf1c05 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:30:28.911633 3627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:28.902865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.3:40711 (local address 127.29.106.1:42425)
0811 22:30:28.905060 (+ 2195us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:28.905079 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:28.905272 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:28.905580 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:28.905585 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:28.905613 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:30:28.905990 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:28.906002 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:28.907062 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:28.907068 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:28.908958 (+ 1890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:28.908972 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:28.910826 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:28.910832 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:28.911005 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:28.911038 (+ 33us) client_negotiation.cc:769] Sending connection context
0811 22:30:28.911146 (+ 108us) client_negotiation.cc:240] Negotiation successful
0811 22:30:28.911313 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2063,"thread_start_us":2207,"threads_started":1}
I20260811 22:30:28.912214 3477 raft_consensus.cc:2454] T 763f84e91fa24badbf64ee2f460a91e3 P b26c21784af645328c1d38a11a7bcbf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d33cf74da520476b999edc281500ca1a in term 1.
I20260811 22:30:28.912230 2996 raft_consensus.cc:2454] T 763f84e91fa24badbf64ee2f460a91e3 P 85a2a18986c042cc9b68076059cf1c05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d33cf74da520476b999edc281500ca1a in term 1.
I20260811 22:30:28.912298 3596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:28.903354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:42425 (local address 127.29.106.3:40711)
0811 22:30:28.903505 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:28.903510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:28.905161 (+ 1651us) server_negotiation.cc:407] Connection header received
0811 22:30:28.905395 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:28.905400 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:28.905466 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:28.905542 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:30:28.906120 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:28.906932 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:28.909168 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:28.910445 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:28.910587 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:28.910650 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:28.911850 (+ 1200us) server_negotiation.cc:299] Negotiation successful
0811 22:30:28.911962 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260811 22:30:28.912549 3626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:28.903249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.2:40789 (local address 127.29.106.1:56489)
0811 22:30:28.905391 (+ 2142us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:28.905408 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 22:30:28.905531 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:30:28.905877 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:30:28.905882 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:30:28.905907 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:30:28.906329 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:28.906342 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:28.907697 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:28.907702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:28.909740 (+ 2038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:30:28.909758 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:30:28.911440 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:30:28.911446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:30:28.911598 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:28.911633 (+ 35us) client_negotiation.cc:769] Sending connection context
0811 22:30:28.912179 (+ 546us) client_negotiation.cc:240] Negotiation successful
0811 22:30:28.912303 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2014,"thread_start_us":1453,"threads_started":1}
I20260811 22:30:28.912117 3597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:28.903501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.106.1:56489 (local address 127.29.106.2:40789)
0811 22:30:28.904213 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:28.904218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:28.905495 (+ 1277us) server_negotiation.cc:407] Connection header received
0811 22:30:28.905707 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:28.905712 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:28.905768 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:28.905841 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:30:28.906502 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:28.907539 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:28.909948 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:28.911068 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:28.911188 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:28.911250 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:28.911783 (+ 533us) server_negotiation.cc:299] Negotiation successful
0811 22:30:28.911899 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606}
I20260811 22:30:28.912626 3127 leader_election.cc:302] T 763f84e91fa24badbf64ee2f460a91e3 P d33cf74da520476b999edc281500ca1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b26c21784af645328c1d38a11a7bcbf6, d33cf74da520476b999edc281500ca1a; no voters:
I20260811 22:30:28.912964 3623 raft_consensus.cc:2781] T 763f84e91fa24badbf64ee2f460a91e3 P d33cf74da520476b999edc281500ca1a [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:30:28.912943 3476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763f84e91fa24badbf64ee2f460a91e3" candidate_uuid: "85a2a18986c042cc9b68076059cf1c05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b26c21784af645328c1d38a11a7bcbf6" is_pre_election: true
I20260811 22:30:28.913321 3476 raft_consensus.cc:2379] T 763f84e91fa24badbf64ee2f460a91e3 P b26c21784af645328c1d38a11a7bcbf6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 85a2a18986c042cc9b68076059cf1c05 in current term 1: Already voted for candidate d33cf74da520476b999edc281500ca1a in this term.
I20260811 22:30:28.913429 3623 raft_consensus.cc:686] T 763f84e91fa24badbf64ee2f460a91e3 P d33cf74da520476b999edc281500ca1a [term 1 LEADER]: Becoming Leader. State: Replica: d33cf74da520476b999edc281500ca1a, State: Running, Role: LEADER
I20260811 22:30:28.913547 3241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "763f84e91fa24badbf64ee2f460a91e3" candidate_uuid: "85a2a18986c042cc9b68076059cf1c05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d33cf74da520476b999edc281500ca1a" is_pre_election: true
I20260811 22:30:28.913980 3623 consensus_queue.cc:237] T 763f84e91fa24badbf64ee2f460a91e3 P d33cf74da520476b999edc281500ca1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b26c21784af645328c1d38a11a7bcbf6" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 40711 } } peers { permanent_uuid: "d33cf74da520476b999edc281500ca1a" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 40789 } } peers { permanent_uuid: "85a2a18986c042cc9b68076059cf1c05" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 41549 } }
I20260811 22:30:28.914638 2872 leader_election.cc:302] T 763f84e91fa24badbf64ee2f460a91e3 P 85a2a18986c042cc9b68076059cf1c05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 85a2a18986c042cc9b68076059cf1c05; no voters: b26c21784af645328c1d38a11a7bcbf6, d33cf74da520476b999edc281500ca1a
I20260811 22:30:28.915004 3616 raft_consensus.cc:2726] T 763f84e91fa24badbf64ee2f460a91e3 P 85a2a18986c042cc9b68076059cf1c05 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260811 22:30:28.918567 2727 catalog_manager.cc:5495] T 763f84e91fa24badbf64ee2f460a91e3 P d33cf74da520476b999edc281500ca1a reported cstate change: term changed from 0 to 1, leader changed from <none> to d33cf74da520476b999edc281500ca1a (127.29.106.2). New cstate: current_term: 1 leader_uuid: "d33cf74da520476b999edc281500ca1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b26c21784af645328c1d38a11a7bcbf6" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 40711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d33cf74da520476b999edc281500ca1a" member_type: VOTER last_known_addr { host: "127.29.106.2" port: 40789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85a2a18986c042cc9b68076059cf1c05" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 41549 } health_report { overall_health: UNKNOWN } } }
I20260811 22:30:28.969383 3597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:28.964101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57300 (local address 127.29.106.2:40789)
0811 22:30:28.964310 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:28.964316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:28.964429 (+ 113us) server_negotiation.cc:407] Connection header received
0811 22:30:28.964643 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:28.964647 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:28.964740 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:28.964863 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:30:28.965242 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:28.965954 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:28.967335 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:28.967763 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:28.967871 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:28.968360 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:30:28.968827 (+ 467us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:30:28.968966 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:28.969045 (+ 79us) server_negotiation.cc:299] Negotiation successful
0811 22:30:28.969119 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260811 22:30:28.974849 3570 tablet.cc:2369] T 763f84e91fa24badbf64ee2f460a91e3 P b26c21784af645328c1d38a11a7bcbf6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:30:28.977065 3476 raft_consensus.cc:1261] T 763f84e91fa24badbf64ee2f460a91e3 P b26c21784af645328c1d38a11a7bcbf6 [term 1 FOLLOWER]: Refusing update from remote peer d33cf74da520476b999edc281500ca1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:30:28.977073 2996 raft_consensus.cc:1261] T 763f84e91fa24badbf64ee2f460a91e3 P 85a2a18986c042cc9b68076059cf1c05 [term 1 FOLLOWER]: Refusing update from remote peer d33cf74da520476b999edc281500ca1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:30:28.977898 3623 consensus_queue.cc:1035] T 763f84e91fa24badbf64ee2f460a91e3 P d33cf74da520476b999edc281500ca1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b26c21784af645328c1d38a11a7bcbf6" member_type: VOTER last_known_addr { host: "127.29.106.3" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:30:28.978189 3632 consensus_queue.cc:1035] T 763f84e91fa24badbf64ee2f460a91e3 P d33cf74da520476b999edc281500ca1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "85a2a18986c042cc9b68076059cf1c05" member_type: VOTER last_known_addr { host: "127.29.106.1" port: 41549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:30:28.989365 3638 mvcc.cc:205] Tried to move back new op lower bound from 7317452509080805376 to 7317452508835786752. Current Snapshot: MvccSnapshot[applied={T|T < 7317452509080805376}]
I20260811 22:30:29.000486 2724 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:30:29.000762 2724 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:30:29.004267 2724 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eeb5555961194908afafba3f73592c53: Sending DeleteTablet for 3 replicas of tablet 763f84e91fa24badbf64ee2f460a91e3
I20260811 22:30:29.005321 3436 tablet_service.cc:1514] Processing DeleteTablet for tablet 763f84e91fa24badbf64ee2f460a91e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:30:28 UTC) from {username='slave'} at 127.0.0.1:56316
I20260811 22:30:29.005373 2958 tablet_service.cc:1514] Processing DeleteTablet for tablet 763f84e91fa24badbf64ee2f460a91e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:30:28 UTC) from {username='slave'} at 127.0.0.1:40836
I20260811 22:30:29.005502 3198 tablet_service.cc:1514] Processing DeleteTablet for tablet 763f84e91fa24badbf64ee2f460a91e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:30:28 UTC) from {username='slave'} at 127.0.0.1:57298
I20260811 22:30:29.008646 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 2844
I20260811 22:30:29.032047 3672 tablet_replica.cc:330] T 763f84e91fa24badbf64ee2f460a91e3 P d33cf74da520476b999edc281500ca1a: stopping tablet replica
I20260811 22:30:29.032545 3672 raft_consensus.cc:2229] T 763f84e91fa24badbf64ee2f460a91e3 P d33cf74da520476b999edc281500ca1a [term 1 LEADER]: Raft consensus shutting down.
I20260811 22:30:29.033289 3672 raft_consensus.cc:2258] T 763f84e91fa24badbf64ee2f460a91e3 P d33cf74da520476b999edc281500ca1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:30:29.034192 3671 tablet_replica.cc:330] T 763f84e91fa24badbf64ee2f460a91e3 P b26c21784af645328c1d38a11a7bcbf6: stopping tablet replica
I20260811 22:30:29.034577 3671 raft_consensus.cc:2229] T 763f84e91fa24badbf64ee2f460a91e3 P b26c21784af645328c1d38a11a7bcbf6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:30:29.035064 3671 raft_consensus.cc:2258] T 763f84e91fa24badbf64ee2f460a91e3 P b26c21784af645328c1d38a11a7bcbf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:30:29.036999 3672 ts_tablet_manager.cc:1905] T 763f84e91fa24badbf64ee2f460a91e3 P d33cf74da520476b999edc281500ca1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:30:29.041867 3600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:29.009630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40844 (local address 127.29.106.1:41549)
0811 22:30:29.009828 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:29.009835 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:29.031223 (+ 21388us) server_negotiation.cc:407] Connection header received
0811 22:30:29.031431 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:29.031437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:29.031585 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:29.031691 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:29.032352 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:29.033591 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:29.035152 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:29.035914 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:29.036046 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:29.040083 (+ 4037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:29.040095 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:29.040114 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:30:29.040881 (+ 767us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:29.040946 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:30:29.040973 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:29.041069 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:29.041319 (+ 250us) server_negotiation.cc:299] Negotiation successful
0811 22:30:29.041453 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260811 22:30:29.043077 3670 tablet_replica.cc:330] T 763f84e91fa24badbf64ee2f460a91e3 P 85a2a18986c042cc9b68076059cf1c05: stopping tablet replica
I20260811 22:30:29.043303 3077 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:30:29.044901 3671 ts_tablet_manager.cc:1905] T 763f84e91fa24badbf64ee2f460a91e3 P b26c21784af645328c1d38a11a7bcbf6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:30:29.049945 3672 ts_tablet_manager.cc:1918] T 763f84e91fa24badbf64ee2f460a91e3 P d33cf74da520476b999edc281500ca1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:30:29.050489 3672 log.cc:1196] T 763f84e91fa24badbf64ee2f460a91e3 P d33cf74da520476b999edc281500ca1a: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-1/wal/wals/763f84e91fa24badbf64ee2f460a91e3
I20260811 22:30:29.051183 3672 ts_tablet_manager.cc:1939] T 763f84e91fa24badbf64ee2f460a91e3 P d33cf74da520476b999edc281500ca1a: Deleting consensus metadata
I20260811 22:30:29.051870 3671 ts_tablet_manager.cc:1918] T 763f84e91fa24badbf64ee2f460a91e3 P b26c21784af645328c1d38a11a7bcbf6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:30:29.052098 3671 log.cc:1196] T 763f84e91fa24badbf64ee2f460a91e3 P b26c21784af645328c1d38a11a7bcbf6: Deleting WAL directory at /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786487105757681-30120-0/minicluster-data/ts-2/wal/wals/763f84e91fa24badbf64ee2f460a91e3
I20260811 22:30:29.052685 3671 ts_tablet_manager.cc:1939] T 763f84e91fa24badbf64ee2f460a91e3 P b26c21784af645328c1d38a11a7bcbf6: Deleting consensus metadata
I20260811 22:30:29.053117 2699 catalog_manager.cc:4841] TS d33cf74da520476b999edc281500ca1a (127.29.106.2:40789): tablet 763f84e91fa24badbf64ee2f460a91e3 (table test-table [id=59fd38f7773f4e6e8fd4ffca67e05114]) successfully deleted
I20260811 22:30:29.055070 2701 catalog_manager.cc:4841] TS b26c21784af645328c1d38a11a7bcbf6 (127.29.106.3:40711): tablet 763f84e91fa24badbf64ee2f460a91e3 (table test-table [id=59fd38f7773f4e6e8fd4ffca67e05114]) successfully deleted
I20260811 22:30:29.043520 3670 raft_consensus.cc:2229] T 763f84e91fa24badbf64ee2f460a91e3 P 85a2a18986c042cc9b68076059cf1c05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:30:29.142242 3670 raft_consensus.cc:2258] T 763f84e91fa24badbf64ee2f460a91e3 P 85a2a18986c042cc9b68076059cf1c05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:30:29.142343 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 2844
I20260811 22:30:29.169909 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 3091
W20260811 22:30:29.170058 2699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.106.1:41549 (error 108)
I20260811 22:30:29.172451 3595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:29.171857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:41549 (local address 127.0.0.1:40854)
0811 22:30:29.172107 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:29.172257 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.1:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
W20260811 22:30:29.172727 2699 catalog_manager.cc:4584] TS 85a2a18986c042cc9b68076059cf1c05 (127.29.106.1:41549): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 763f84e91fa24badbf64ee2f460a91e3: Network error: Client connection negotiation failed: client connection to 127.29.106.1:41549: connect: Connection refused (error 111)
I20260811 22:30:29.177511 3597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:29.170659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57312 (local address 127.29.106.2:40789)
0811 22:30:29.171333 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:29.171339 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:29.171360 (+ 21us) server_negotiation.cc:407] Connection header received
0811 22:30:29.171452 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:29.171458 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:29.171595 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:29.171710 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:29.172214 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:29.173149 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:29.174398 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:29.174855 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:29.174990 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:29.176148 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:29.176157 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:29.176174 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:30:29.176766 (+ 592us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:29.176837 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:30:29.176859 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:29.176953 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:29.177104 (+ 151us) server_negotiation.cc:299] Negotiation successful
0811 22:30:29.177206 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504}
I20260811 22:30:29.178282 3315 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:30:29.226269 3595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:29.225760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:41549 (local address 127.0.0.1:40862)
0811 22:30:29.226001 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:29.226100 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.1:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260811 22:30:29.227371 3595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:29.226986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:41549 (local address 127.0.0.1:40872)
0811 22:30:29.227126 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:29.227201 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.1:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260811 22:30:29.288244 3595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:29.287475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:41549 (local address 127.0.0.1:40888)
0811 22:30:29.287960 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:29.288062 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.1:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356}
I20260811 22:30:29.289309 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 3091
I20260811 22:30:29.289737 3595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:29.289216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:41549 (local address 127.0.0.1:40904)
0811 22:30:29.289493 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:29.289588 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.1:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
I20260811 22:30:29.311090 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 3327
I20260811 22:30:29.318336 3596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:29.311836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56328 (local address 127.29.106.3:40711)
0811 22:30:29.311997 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:29.312003 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:29.312063 (+ 60us) server_negotiation.cc:407] Connection header received
0811 22:30:29.312210 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:29.312214 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:29.312301 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:29.312396 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:29.312853 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:29.313559 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:29.315419 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:29.315826 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:29.315930 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:29.317015 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:29.317027 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:29.317044 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:30:29.317615 (+ 571us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:29.317665 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:30:29.317686 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:29.317775 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:29.317950 (+ 175us) server_negotiation.cc:299] Negotiation successful
0811 22:30:29.318039 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260811 22:30:29.318918 3561 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:30:29.370105 3595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:29.369625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:41549 (local address 127.0.0.1:40912)
0811 22:30:29.369816 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:29.369927 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.1:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260811 22:30:29.371093 3595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:29.370684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:41549 (local address 127.0.0.1:40916)
0811 22:30:29.370874 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:29.370944 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.1:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260811 22:30:29.409817 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 3327
I20260811 22:30:29.437269 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 2664
I20260811 22:30:29.438349 2794 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:30:29.549386 3595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:29.548856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.1:41549 (local address 127.0.0.1:40932)
0811 22:30:29.549073 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0811 22:30:29.549175 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.1:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260811 22:30:29.549441 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 2664
2026-08-11T22:30:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2750 ms)
[----------] 6 tests from EncryptionPolicyTest (22160 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786487105757681-30120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:30:29 dist-test-slave-gjrz krb5kdc[3692](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:30:29 dist-test-slave-gjrz krb5kdc[3692](info): set up 2 sockets
Aug 11 22:30:29 dist-test-slave-gjrz krb5kdc[3692](info): commencing operation
krb5kdc: starting...
W20260811 22:30:30.829648 30120 mini_kdc.cc:121] Time spent starting KDC: real 1.227s user 0.001s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Aug 11 22:30:31 dist-test-slave-gjrz krb5kdc[3692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:30:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:30:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786487105757681-30120-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786487105757681-30120-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 22:30:31.104864 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46673
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786487105757681-30120-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:30:31.427668 3717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:30:31.427960 3717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:30:31.428105 3717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:30:31.435089 3717 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:30:31.435204 3717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:30:31.435302 3717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:30:31.435403 3717 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:30:31.435462 3717 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:30:31.444802 3717 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46673
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786487105757681-30120-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:30:31.445946 3717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:30:31.447149 3717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:30:31.458145 3725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:31.458120 3727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:31.458179 3722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:30:31.458907 3717 server_base.cc:1034] running on GCE node
I20260811 22:30:31.459662 3717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:30:31.460965 3717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:30:31.462158 3717 hybrid_clock.cc:648] HybridClock initialized: now 1786487431462111 us; error 40 us; skew 500 ppm
I20260811 22:30:31.462488 3717 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:30:31 dist-test-slave-gjrz krb5kdc[3692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:30:31.465410 3717 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 22:30:31.466809 3717 webserver.cc:460] Webserver started at http://127.0.0.1:46139/ using document root <none> and password file <none>
I20260811 22:30:31.467494 3717 fs_manager.cc:359] Metadata directory not provided
I20260811 22:30:31.467625 3717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:30:31.467895 3717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:30:31.470757 3717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "b993f5f891fc4493af11771e2ff5a72b"
format_stamp: "Formatted at 2026-08-11 22:30:31 on dist-test-slave-gjrz"
server_key: "189c29ddc8328e610f0ba130bb471c62"
server_key_iv: "2784ba563a7e96a48bcdb09c851b4e1f"
server_key_version: "encryptionkey@0"
I20260811 22:30:31.471553 3717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "b993f5f891fc4493af11771e2ff5a72b"
format_stamp: "Formatted at 2026-08-11 22:30:31 on dist-test-slave-gjrz"
server_key: "189c29ddc8328e610f0ba130bb471c62"
server_key_iv: "2784ba563a7e96a48bcdb09c851b4e1f"
server_key_version: "encryptionkey@0"
I20260811 22:30:31.476343 3717 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 22:30:31.479530 3742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:31.480728 3717 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260811 22:30:31.480887 3717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "b993f5f891fc4493af11771e2ff5a72b"
format_stamp: "Formatted at 2026-08-11 22:30:31 on dist-test-slave-gjrz"
server_key: "189c29ddc8328e610f0ba130bb471c62"
server_key_iv: "2784ba563a7e96a48bcdb09c851b4e1f"
server_key_version: "encryptionkey@0"
I20260811 22:30:31.481079 3717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:30:31.507140 3717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:30:31.510923 3717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:30:31.511168 3717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:30:31.528414 3717 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46673
I20260811 22:30:31.528481 3846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46673 every 8 connection(s)
I20260811 22:30:31.529623 3717 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:30:31.533154 3848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:30:31.537571 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 3717
I20260811 22:30:31.537806 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:30:31.538631 30120 external_mini_cluster.cc:1428] Setting key 32b603f7e218a44b25218b1a916d3648
I20260811 22:30:31.544209 3848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b993f5f891fc4493af11771e2ff5a72b: Bootstrap starting.
I20260811 22:30:31.547668 3848 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b993f5f891fc4493af11771e2ff5a72b: Neither blocks nor log segments found. Creating new log.
I20260811 22:30:31.548858 3848 log.cc:824] T 00000000000000000000000000000000 P b993f5f891fc4493af11771e2ff5a72b: Log is configured to *not* fsync() on all Append() calls
I20260811 22:30:31.551545 3848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b993f5f891fc4493af11771e2ff5a72b: No bootstrap required, opened a new log
Aug 11 22:30:31 dist-test-slave-gjrz krb5kdc[3692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260811 22:30:31.561323 3848 raft_consensus.cc:348] T 00000000000000000000000000000000 P b993f5f891fc4493af11771e2ff5a72b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b993f5f891fc4493af11771e2ff5a72b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46673 } }
I20260811 22:30:31.561750 3848 raft_consensus.cc:374] T 00000000000000000000000000000000 P b993f5f891fc4493af11771e2ff5a72b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:30:31.561846 3853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:31.540503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50575 (local address 127.0.0.1:46673)
0811 22:30:31.541278 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:31.541289 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:31.541328 (+ 39us) server_negotiation.cc:407] Connection header received
0811 22:30:31.543054 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:31.543075 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:31.543472 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:31.543772 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:31.545507 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:31.546864 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:31.548684 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:31.549493 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:31.549684 (+ 191us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:31.553521 (+ 3837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:31.553531 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:31.553548 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:30:31.553594 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:31.557477 (+ 3883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:31.558201 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:31.558210 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:31.558218 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:31.558446 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:31.558868 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:31.558873 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:31.558876 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:31.559603 (+ 727us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:31.559947 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:31.560269 (+ 322us) server_negotiation.cc:299] Negotiation successful
0811 22:30:31.560599 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":351,"threads_started":1}
I20260811 22:30:31.561893 3848 raft_consensus.cc:729] T 00000000000000000000000000000000 P b993f5f891fc4493af11771e2ff5a72b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b993f5f891fc4493af11771e2ff5a72b, State: Initialized, Role: FOLLOWER
I20260811 22:30:31.562479 3848 consensus_queue.cc:260] T 00000000000000000000000000000000 P b993f5f891fc4493af11771e2ff5a72b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b993f5f891fc4493af11771e2ff5a72b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46673 } }
I20260811 22:30:31.562719 3848 raft_consensus.cc:388] T 00000000000000000000000000000000 P b993f5f891fc4493af11771e2ff5a72b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:30:31.562860 3848 raft_consensus.cc:482] T 00000000000000000000000000000000 P b993f5f891fc4493af11771e2ff5a72b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:30:31.563035 3848 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b993f5f891fc4493af11771e2ff5a72b [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:30:31.565948 30120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:30:31.565968 3848 raft_consensus.cc:504] T 00000000000000000000000000000000 P b993f5f891fc4493af11771e2ff5a72b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b993f5f891fc4493af11771e2ff5a72b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46673 } }
I20260811 22:30:31.566501 3848 leader_election.cc:302] T 00000000000000000000000000000000 P b993f5f891fc4493af11771e2ff5a72b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b993f5f891fc4493af11771e2ff5a72b; no voters:
I20260811 22:30:31.567119 3848 leader_election.cc:288] T 00000000000000000000000000000000 P b993f5f891fc4493af11771e2ff5a72b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:30:31.567202 3858 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b993f5f891fc4493af11771e2ff5a72b [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:30:31.569128 3848 sys_catalog.cc:564] T 00000000000000000000000000000000 P b993f5f891fc4493af11771e2ff5a72b [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:30:31.571588 3858 raft_consensus.cc:686] T 00000000000000000000000000000000 P b993f5f891fc4493af11771e2ff5a72b [term 1 LEADER]: Becoming Leader. State: Replica: b993f5f891fc4493af11771e2ff5a72b, State: Running, Role: LEADER
I20260811 22:30:31.572574 3858 consensus_queue.cc:237] T 00000000000000000000000000000000 P b993f5f891fc4493af11771e2ff5a72b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b993f5f891fc4493af11771e2ff5a72b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46673 } }
I20260811 22:30:31.575418 3859 sys_catalog.cc:455] T 00000000000000000000000000000000 P b993f5f891fc4493af11771e2ff5a72b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b993f5f891fc4493af11771e2ff5a72b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b993f5f891fc4493af11771e2ff5a72b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46673 } } }
I20260811 22:30:31.575776 3859 sys_catalog.cc:458] T 00000000000000000000000000000000 P b993f5f891fc4493af11771e2ff5a72b [sys.catalog]: This master's current role is: LEADER
I20260811 22:30:31.575220 3860 sys_catalog.cc:455] T 00000000000000000000000000000000 P b993f5f891fc4493af11771e2ff5a72b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b993f5f891fc4493af11771e2ff5a72b. Latest consensus state: current_term: 1 leader_uuid: "b993f5f891fc4493af11771e2ff5a72b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b993f5f891fc4493af11771e2ff5a72b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46673 } } }
I20260811 22:30:31.576411 3860 sys_catalog.cc:458] T 00000000000000000000000000000000 P b993f5f891fc4493af11771e2ff5a72b [sys.catalog]: This master's current role is: LEADER
I20260811 22:30:31.581280 3886 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:30:31.586706 3886 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:30:31.592031 3853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:31.572373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36907 (local address 127.0.0.1:46673)
0811 22:30:31.573729 (+ 1356us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:31.573733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:31.573754 (+ 21us) server_negotiation.cc:407] Connection header received
0811 22:30:31.573799 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:31.573804 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:31.573869 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:31.573958 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:31.576649 (+ 2691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:31.577833 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:31.579753 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:31.580493 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:31.580612 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:31.583786 (+ 3174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:31.583792 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:31.583797 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:30:31.583835 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:31.587932 (+ 4097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:31.588941 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:31.588946 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:31.588950 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:31.589118 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:31.589708 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:31.589713 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:31.589717 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:31.590140 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:31.591301 (+ 1161us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:31.591482 (+ 181us) server_negotiation.cc:299] Negotiation successful
0811 22:30:31.591667 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1269}
I20260811 22:30:31.596295 3886 catalog_manager.cc:1347] Generated new cluster ID: a31511ec33ea4c93a30905c8c3aca266
I20260811 22:30:31.596410 3886 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:30:31.611984 3886 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:30:31.613127 3886 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:30:31.621487 3886 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b993f5f891fc4493af11771e2ff5a72b: Generated new TSK 0
I20260811 22:30:31.622418 3886 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:30:31.630666 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 3717
I20260811 22:30:31.632018 3838 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:30:31.721145 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 3717
W20260811 22:30:31.747219 3689 connection.cc:535] client connection to 127.0.0.1:46673 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46673): Connection reset by peer (error 104)
2026-08-11T22:30:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2164 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786487105757681-30120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:30:31 dist-test-slave-gjrz krb5kdc[3910](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:30:31 dist-test-slave-gjrz krb5kdc[3910](info): set up 2 sockets
Aug 11 22:30:31 dist-test-slave-gjrz krb5kdc[3910](info): commencing operation
krb5kdc: starting...
W20260811 22:30:32.983850 30120 mini_kdc.cc:121] Time spent starting KDC: real 1.218s user 0.001s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Aug 11 22:30:33 dist-test-slave-gjrz krb5kdc[3910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:30:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:30:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786487105757681-30120-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786487105757681-30120-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 22:30:33.262202 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46251
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786487105757681-30120-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:30:33.596273 3930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:30:33.596546 3930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:30:33.596642 3930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:30:33.603744 3930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:30:33.603852 3930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:30:33.603905 3930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:30:33.603967 3930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:30:33.604012 3930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:30:33.613511 3930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46251
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786487105757681-30120-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:30:33.614643 3930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:30:33.615895 3930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:30:33.627010 3945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:33.627116 3941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:33.627396 3940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:30:33.628198 3930 server_base.cc:1034] running on GCE node
I20260811 22:30:33.628890 3930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:30:33.630208 3930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:30:33.631399 3930 hybrid_clock.cc:648] HybridClock initialized: now 1786487433631356 us; error 36 us; skew 500 ppm
I20260811 22:30:33.631737 3930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:30:33 dist-test-slave-gjrz krb5kdc[3910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:30:33.634828 3930 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 22:30:33.636276 3930 webserver.cc:460] Webserver started at http://127.0.0.1:44687/ using document root <none> and password file <none>
I20260811 22:30:33.636898 3930 fs_manager.cc:359] Metadata directory not provided
I20260811 22:30:33.637007 3930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:30:33.637256 3930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:30:33.640192 3930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "8ba960aa1c544824ae936470cd701e3a"
format_stamp: "Formatted at 2026-08-11 22:30:33 on dist-test-slave-gjrz"
server_key: "bf4efe891519645eee5561e4a1604e90"
server_key_iv: "b53735aaa4ee2477a593c5886c683343"
server_key_version: "encryptionkey@0"
I20260811 22:30:33.640925 3930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "8ba960aa1c544824ae936470cd701e3a"
format_stamp: "Formatted at 2026-08-11 22:30:33 on dist-test-slave-gjrz"
server_key: "bf4efe891519645eee5561e4a1604e90"
server_key_iv: "b53735aaa4ee2477a593c5886c683343"
server_key_version: "encryptionkey@0"
I20260811 22:30:33.645912 3930 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260811 22:30:33.649366 3958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:33.650657 3930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 22:30:33.650804 3930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "8ba960aa1c544824ae936470cd701e3a"
format_stamp: "Formatted at 2026-08-11 22:30:33 on dist-test-slave-gjrz"
server_key: "bf4efe891519645eee5561e4a1604e90"
server_key_iv: "b53735aaa4ee2477a593c5886c683343"
server_key_version: "encryptionkey@0"
I20260811 22:30:33.650966 3930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:30:33.684571 3930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:30:33.688385 3930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:30:33.688623 3930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:30:33.706393 3930 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46251
I20260811 22:30:33.706434 4059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46251 every 8 connection(s)
I20260811 22:30:33.707737 3930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:30:33.711323 4061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:30:33.717391 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 3930
I20260811 22:30:33.717586 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:30:33.718191 30120 external_mini_cluster.cc:1428] Setting key 9564d4a33f334e74c47f4bce8b4a64ba
I20260811 22:30:33.722034 4061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ba960aa1c544824ae936470cd701e3a: Bootstrap starting.
I20260811 22:30:33.725306 4061 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ba960aa1c544824ae936470cd701e3a: Neither blocks nor log segments found. Creating new log.
I20260811 22:30:33.726544 4061 log.cc:824] T 00000000000000000000000000000000 P 8ba960aa1c544824ae936470cd701e3a: Log is configured to *not* fsync() on all Append() calls
I20260811 22:30:33.729990 4061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ba960aa1c544824ae936470cd701e3a: No bootstrap required, opened a new log
I20260811 22:30:33.731904 4064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:33.719910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53047 (local address 127.0.0.1:46251)
0811 22:30:33.720687 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:33.720698 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:33.720739 (+ 41us) server_negotiation.cc:407] Connection header received
0811 22:30:33.722410 (+ 1671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:33.722432 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:33.722796 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:33.723097 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:33.723744 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:33.725053 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:33.726933 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:33.728036 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:33.728167 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:33.729585 (+ 1418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:33.729592 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:33.729607 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:30:33.730162 (+ 555us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:33.730246 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:30:33.730271 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:33.730393 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:33.730701 (+ 308us) server_negotiation.cc:299] Negotiation successful
0811 22:30:33.730879 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":368,"threads_started":1}
I20260811 22:30:33.735617 30120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:30:33.738905 4061 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ba960aa1c544824ae936470cd701e3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ba960aa1c544824ae936470cd701e3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46251 } }
I20260811 22:30:33.739415 4061 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ba960aa1c544824ae936470cd701e3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:30:33.739540 4061 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ba960aa1c544824ae936470cd701e3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ba960aa1c544824ae936470cd701e3a, State: Initialized, Role: FOLLOWER
I20260811 22:30:33.740193 4061 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ba960aa1c544824ae936470cd701e3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ba960aa1c544824ae936470cd701e3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46251 } }
I20260811 22:30:33.740484 4061 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ba960aa1c544824ae936470cd701e3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:30:33.740620 4061 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ba960aa1c544824ae936470cd701e3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:30:33.740818 4061 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ba960aa1c544824ae936470cd701e3a [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:30:33.743680 4061 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ba960aa1c544824ae936470cd701e3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ba960aa1c544824ae936470cd701e3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46251 } }
I20260811 22:30:33.744285 4061 leader_election.cc:302] T 00000000000000000000000000000000 P 8ba960aa1c544824ae936470cd701e3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ba960aa1c544824ae936470cd701e3a; no voters:
I20260811 22:30:33.744910 4061 leader_election.cc:288] T 00000000000000000000000000000000 P 8ba960aa1c544824ae936470cd701e3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:30:33.745002 4079 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ba960aa1c544824ae936470cd701e3a [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:30:33.746991 4061 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ba960aa1c544824ae936470cd701e3a [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:30:33.747565 4079 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ba960aa1c544824ae936470cd701e3a [term 1 LEADER]: Becoming Leader. State: Replica: 8ba960aa1c544824ae936470cd701e3a, State: Running, Role: LEADER
I20260811 22:30:33.748162 4079 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ba960aa1c544824ae936470cd701e3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ba960aa1c544824ae936470cd701e3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46251 } }
I20260811 22:30:33.748708 4064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:33.740942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40129 (local address 127.0.0.1:46251)
0811 22:30:33.741126 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:33.741131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:33.741684 (+ 553us) server_negotiation.cc:407] Connection header received
0811 22:30:33.741851 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:33.741854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:33.741926 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:33.742022 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:33.742566 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:33.743364 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:33.745271 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:33.746048 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:33.746177 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:33.747772 (+ 1595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:33.747778 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:33.747783 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:30:33.747807 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:33.747882 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:30:33.747903 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:33.748014 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:33.748279 (+ 265us) server_negotiation.cc:299] Negotiation successful
0811 22:30:33.748345 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260811 22:30:33.752858 4081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ba960aa1c544824ae936470cd701e3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ba960aa1c544824ae936470cd701e3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ba960aa1c544824ae936470cd701e3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46251 } } }
I20260811 22:30:33.753280 4081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ba960aa1c544824ae936470cd701e3a [sys.catalog]: This master's current role is: LEADER
I20260811 22:30:33.753731 4082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ba960aa1c544824ae936470cd701e3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ba960aa1c544824ae936470cd701e3a. Latest consensus state: current_term: 1 leader_uuid: "8ba960aa1c544824ae936470cd701e3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ba960aa1c544824ae936470cd701e3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46251 } } }
I20260811 22:30:33.754017 4082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ba960aa1c544824ae936470cd701e3a [sys.catalog]: This master's current role is: LEADER
I20260811 22:30:33.757102 4100 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:30:33.761217 4100 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:30:33.769593 4100 catalog_manager.cc:1347] Generated new cluster ID: fe264474a38c4782899d25deffef2fb7
I20260811 22:30:33.769697 4100 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:30:33.779124 4100 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:30:33.780857 4100 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:30:33.792513 4100 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ba960aa1c544824ae936470cd701e3a: Generated new TSK 0
I20260811 22:30:33.793207 4100 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:30:33.801658 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 3930
W20260811 22:30:33.802708 4047 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53047
W20260811 22:30:33.803434 30120 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 3930: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 22:30:33.803645 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 3930
W20260811 22:30:33.836210 3904 connection.cc:535] client connection to 127.0.0.1:46251 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46251): Connection reset by peer (error 104)
2026-08-11T22:30:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2088 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786487105757681-30120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:30:33 dist-test-slave-gjrz krb5kdc[4119](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:30:33 dist-test-slave-gjrz krb5kdc[4119](info): set up 2 sockets
Aug 11 22:30:33 dist-test-slave-gjrz krb5kdc[4119](info): commencing operation
krb5kdc: starting...
W20260811 22:30:35.084131 30120 mini_kdc.cc:121] Time spent starting KDC: real 1.229s user 0.000s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Aug 11 22:30:35 dist-test-slave-gjrz krb5kdc[4119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:30:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:30:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786487105757681-30120-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786487105757681-30120-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 22:30:35.362679 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44841
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786487105757681-30120-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:30:35.692844 4141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:30:35.693109 4141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:30:35.693208 4141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:30:35.700155 4141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:30:35.700245 4141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:30:35.700297 4141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:30:35.700367 4141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:30:35.700412 4141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:30:35.709653 4141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44841
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786487105757681-30120-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:30:35.710716 4141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:30:35.711970 4141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:30:35.722946 4152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:35.722875 4154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:35.722913 4156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:30:35.723611 4141 server_base.cc:1034] running on GCE node
I20260811 22:30:35.724269 4141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:30:35.725436 4141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:30:35.726640 4141 hybrid_clock.cc:648] HybridClock initialized: now 1786487435726600 us; error 38 us; skew 500 ppm
I20260811 22:30:35.726970 4141 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:30:35 dist-test-slave-gjrz krb5kdc[4119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487435, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:30:35.736016 4141 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 22:30:35.737372 4141 webserver.cc:460] Webserver started at http://127.0.0.1:34137/ using document root <none> and password file <none>
I20260811 22:30:35.737982 4141 fs_manager.cc:359] Metadata directory not provided
I20260811 22:30:35.738080 4141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:30:35.738323 4141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:30:35.741137 4141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "781b2255e4e94a249cb0ed6a52bc0959"
format_stamp: "Formatted at 2026-08-11 22:30:35 on dist-test-slave-gjrz"
server_key: "3522f45eb09542b6b9ad57ff287b24da"
server_key_iv: "09f3e1a1675ecd5b6d4963b4a75bc92b"
server_key_version: "encryptionkey@0"
I20260811 22:30:35.741818 4141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "781b2255e4e94a249cb0ed6a52bc0959"
format_stamp: "Formatted at 2026-08-11 22:30:35 on dist-test-slave-gjrz"
server_key: "3522f45eb09542b6b9ad57ff287b24da"
server_key_iv: "09f3e1a1675ecd5b6d4963b4a75bc92b"
server_key_version: "encryptionkey@0"
I20260811 22:30:35.746516 4141 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260811 22:30:35.749711 4165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:35.750910 4141 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260811 22:30:35.751055 4141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "781b2255e4e94a249cb0ed6a52bc0959"
format_stamp: "Formatted at 2026-08-11 22:30:35 on dist-test-slave-gjrz"
server_key: "3522f45eb09542b6b9ad57ff287b24da"
server_key_iv: "09f3e1a1675ecd5b6d4963b4a75bc92b"
server_key_version: "encryptionkey@0"
I20260811 22:30:35.751233 4141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:30:35.756630 4141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:30:35.760182 4141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:30:35.760442 4141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:30:35.777663 4141 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44841
I20260811 22:30:35.777719 4271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44841 every 8 connection(s)
I20260811 22:30:35.778931 4141 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:30:35.781061 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 4141
I20260811 22:30:35.781263 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:30:35.781795 30120 external_mini_cluster.cc:1428] Setting key 1f08de749abf689c93877dd502510ef0
I20260811 22:30:35.782892 4275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:30:35.789036 4281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:35.783540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42309 (local address 127.0.0.1:44841)
0811 22:30:35.784274 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:35.784284 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:35.784323 (+ 39us) server_negotiation.cc:407] Connection header received
0811 22:30:35.785827 (+ 1504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:35.785847 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:35.786172 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:35.786521 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:35.787548 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:35.787555 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:35.787568 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:30:35.787582 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:35.787659 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:30:35.787679 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:35.787895 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:35.788050 (+ 155us) server_negotiation.cc:299] Negotiation successful
0811 22:30:35.788212 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":310,"threads_started":1}
I20260811 22:30:35.792271 30120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:30:35.794565 4275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 781b2255e4e94a249cb0ed6a52bc0959: Bootstrap starting.
I20260811 22:30:35.797760 4275 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 781b2255e4e94a249cb0ed6a52bc0959: Neither blocks nor log segments found. Creating new log.
I20260811 22:30:35.798460 4281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:35.796296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55633 (local address 127.0.0.1:44841)
0811 22:30:35.796467 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:35.796474 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:35.796790 (+ 316us) server_negotiation.cc:407] Connection header received
0811 22:30:35.796940 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:35.796943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:35.796985 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:35.797059 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:35.797856 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:35.797862 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:35.797866 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:30:35.797882 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:35.797936 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:30:35.797954 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:35.798049 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:35.798168 (+ 119us) server_negotiation.cc:299] Negotiation successful
0811 22:30:35.798221 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260811 22:30:35.798918 4275 log.cc:824] T 00000000000000000000000000000000 P 781b2255e4e94a249cb0ed6a52bc0959: Log is configured to *not* fsync() on all Append() calls
I20260811 22:30:35.801365 4275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 781b2255e4e94a249cb0ed6a52bc0959: No bootstrap required, opened a new log
I20260811 22:30:35.808274 4275 raft_consensus.cc:348] T 00000000000000000000000000000000 P 781b2255e4e94a249cb0ed6a52bc0959 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "781b2255e4e94a249cb0ed6a52bc0959" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44841 } }
I20260811 22:30:35.808609 4275 raft_consensus.cc:374] T 00000000000000000000000000000000 P 781b2255e4e94a249cb0ed6a52bc0959 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:30:35.808696 4275 raft_consensus.cc:729] T 00000000000000000000000000000000 P 781b2255e4e94a249cb0ed6a52bc0959 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 781b2255e4e94a249cb0ed6a52bc0959, State: Initialized, Role: FOLLOWER
I20260811 22:30:35.809221 4275 consensus_queue.cc:260] T 00000000000000000000000000000000 P 781b2255e4e94a249cb0ed6a52bc0959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "781b2255e4e94a249cb0ed6a52bc0959" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44841 } }
I20260811 22:30:35.809449 4275 raft_consensus.cc:388] T 00000000000000000000000000000000 P 781b2255e4e94a249cb0ed6a52bc0959 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:30:35.809579 4275 raft_consensus.cc:482] T 00000000000000000000000000000000 P 781b2255e4e94a249cb0ed6a52bc0959 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:30:35.809736 4275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 781b2255e4e94a249cb0ed6a52bc0959 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:30:35.811800 4275 raft_consensus.cc:504] T 00000000000000000000000000000000 P 781b2255e4e94a249cb0ed6a52bc0959 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "781b2255e4e94a249cb0ed6a52bc0959" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44841 } }
I20260811 22:30:35.812232 4275 leader_election.cc:302] T 00000000000000000000000000000000 P 781b2255e4e94a249cb0ed6a52bc0959 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 781b2255e4e94a249cb0ed6a52bc0959; no voters:
I20260811 22:30:35.812736 4275 leader_election.cc:288] T 00000000000000000000000000000000 P 781b2255e4e94a249cb0ed6a52bc0959 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:30:35.812841 4293 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 781b2255e4e94a249cb0ed6a52bc0959 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:30:35.813599 4293 raft_consensus.cc:686] T 00000000000000000000000000000000 P 781b2255e4e94a249cb0ed6a52bc0959 [term 1 LEADER]: Becoming Leader. State: Replica: 781b2255e4e94a249cb0ed6a52bc0959, State: Running, Role: LEADER
I20260811 22:30:35.814168 4293 consensus_queue.cc:237] T 00000000000000000000000000000000 P 781b2255e4e94a249cb0ed6a52bc0959 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "781b2255e4e94a249cb0ed6a52bc0959" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44841 } }
I20260811 22:30:35.814605 4275 sys_catalog.cc:564] T 00000000000000000000000000000000 P 781b2255e4e94a249cb0ed6a52bc0959 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:30:35.816869 4294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 781b2255e4e94a249cb0ed6a52bc0959 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "781b2255e4e94a249cb0ed6a52bc0959" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "781b2255e4e94a249cb0ed6a52bc0959" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44841 } } }
I20260811 22:30:35.816829 4296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 781b2255e4e94a249cb0ed6a52bc0959 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 781b2255e4e94a249cb0ed6a52bc0959. Latest consensus state: current_term: 1 leader_uuid: "781b2255e4e94a249cb0ed6a52bc0959" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "781b2255e4e94a249cb0ed6a52bc0959" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44841 } } }
I20260811 22:30:35.817411 4296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 781b2255e4e94a249cb0ed6a52bc0959 [sys.catalog]: This master's current role is: LEADER
I20260811 22:30:35.817411 4294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 781b2255e4e94a249cb0ed6a52bc0959 [sys.catalog]: This master's current role is: LEADER
I20260811 22:30:35.819676 4302 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:30:35.824244 4302 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:30:35.831626 4302 catalog_manager.cc:1347] Generated new cluster ID: 705bc677e66f48d8b9141f6ea329222a
I20260811 22:30:35.831732 4302 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:30:35.871189 4302 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:30:35.871721 4302 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:30:35.887348 4302 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 781b2255e4e94a249cb0ed6a52bc0959: Generated new TSK 0
I20260811 22:30:35.888115 4302 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:30:35.903607 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 4141
W20260811 22:30:35.904644 4262 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42309
W20260811 22:30:35.905452 30120 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 4141: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 22:30:35.905593 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 4141
2026-08-11T22:30:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2095 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786487105757681-30120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:30:36 dist-test-slave-gjrz krb5kdc[4337](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:30:36 dist-test-slave-gjrz krb5kdc[4337](info): set up 2 sockets
Aug 11 22:30:36 dist-test-slave-gjrz krb5kdc[4337](info): commencing operation
krb5kdc: starting...
W20260811 22:30:37.199400 30120 mini_kdc.cc:121] Time spent starting KDC: real 1.249s user 0.001s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Aug 11 22:30:37 dist-test-slave-gjrz krb5kdc[4337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:30:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:30:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786487105757681-30120-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786487105757681-30120-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 22:30:37.484110 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42709
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786487105757681-30120-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:30:37.813961 4355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:30:37.814215 4355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:30:37.814306 4355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:30:37.821249 4355 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:30:37.821346 4355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:30:37.821391 4355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:30:37.821451 4355 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:30:37.821496 4355 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:30:37.830765 4355 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786487105757681-30120-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:30:37.831885 4355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:30:37.833048 4355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:30:37.843802 4365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:37.843861 4364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:37.843876 4367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:30:37.844807 4355 server_base.cc:1034] running on GCE node
I20260811 22:30:37.845549 4355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:30:37.846814 4355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:30:37.848052 4355 hybrid_clock.cc:648] HybridClock initialized: now 1786487437847992 us; error 41 us; skew 500 ppm
I20260811 22:30:37.848403 4355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:30:37 dist-test-slave-gjrz krb5kdc[4337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:30:37.851503 4355 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 22:30:37.852831 4355 webserver.cc:460] Webserver started at http://127.0.0.1:37085/ using document root <none> and password file <none>
I20260811 22:30:37.853420 4355 fs_manager.cc:359] Metadata directory not provided
I20260811 22:30:37.853519 4355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:30:37.853754 4355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:30:37.856604 4355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "ed48d3555d7147f08a8a07ac9adf0d97"
format_stamp: "Formatted at 2026-08-11 22:30:37 on dist-test-slave-gjrz"
server_key: "3652ddfd25ab8f9ab2938ede22ffd5e5"
server_key_iv: "efb7f7c1fcb6ec99abce8758839cafd5"
server_key_version: "encryptionkey@0"
I20260811 22:30:37.857326 4355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "ed48d3555d7147f08a8a07ac9adf0d97"
format_stamp: "Formatted at 2026-08-11 22:30:37 on dist-test-slave-gjrz"
server_key: "3652ddfd25ab8f9ab2938ede22ffd5e5"
server_key_iv: "efb7f7c1fcb6ec99abce8758839cafd5"
server_key_version: "encryptionkey@0"
I20260811 22:30:37.862210 4355 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260811 22:30:37.865506 4379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:37.866703 4355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260811 22:30:37.866847 4355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "ed48d3555d7147f08a8a07ac9adf0d97"
format_stamp: "Formatted at 2026-08-11 22:30:37 on dist-test-slave-gjrz"
server_key: "3652ddfd25ab8f9ab2938ede22ffd5e5"
server_key_iv: "efb7f7c1fcb6ec99abce8758839cafd5"
server_key_version: "encryptionkey@0"
I20260811 22:30:37.867036 4355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:30:37.882884 4355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:30:37.887048 4355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:30:37.887328 4355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:30:37.904634 4355 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42709
I20260811 22:30:37.904665 4475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42709 every 8 connection(s)
I20260811 22:30:37.905890 4355 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:30:37.909564 4476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:30:37.910933 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 4355
I20260811 22:30:37.911131 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:30:37.911727 30120 external_mini_cluster.cc:1428] Setting key 1c78f7d70f81a5b098b9a4f408d5ffcf
I20260811 22:30:37.923655 4476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed48d3555d7147f08a8a07ac9adf0d97: Bootstrap starting.
Aug 11 22:30:37 dist-test-slave-gjrz krb5kdc[4337](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260811 22:30:37.927472 4476 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed48d3555d7147f08a8a07ac9adf0d97: Neither blocks nor log segments found. Creating new log.
I20260811 22:30:37.928854 4476 log.cc:824] T 00000000000000000000000000000000 P ed48d3555d7147f08a8a07ac9adf0d97: Log is configured to *not* fsync() on all Append() calls
I20260811 22:30:37.931948 4476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed48d3555d7147f08a8a07ac9adf0d97: No bootstrap required, opened a new log
I20260811 22:30:37.934965 4479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:37.913475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49191 (local address 127.0.0.1:42709)
0811 22:30:37.914257 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:37.914268 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:37.914312 (+ 44us) server_negotiation.cc:407] Connection header received
0811 22:30:37.915970 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:37.916000 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:37.916379 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:37.916689 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:37.918496 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:37.919836 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:37.921707 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:37.922378 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:37.922503 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:37.927408 (+ 4905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:37.927416 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:37.927431 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:30:37.927469 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:37.930930 (+ 3461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:37.931766 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:37.931773 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:37.931778 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:37.931965 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:37.932367 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:37.932372 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:37.932374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:37.932883 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:37.933159 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:37.933652 (+ 493us) server_negotiation.cc:299] Negotiation successful
0811 22:30:37.934033 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":359,"threads_started":1}
I20260811 22:30:37.938604 30120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:30:37.940203 4476 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed48d3555d7147f08a8a07ac9adf0d97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed48d3555d7147f08a8a07ac9adf0d97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42709 } }
I20260811 22:30:37.940521 4476 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed48d3555d7147f08a8a07ac9adf0d97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:30:37.940606 4476 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed48d3555d7147f08a8a07ac9adf0d97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed48d3555d7147f08a8a07ac9adf0d97, State: Initialized, Role: FOLLOWER
I20260811 22:30:37.941170 4476 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed48d3555d7147f08a8a07ac9adf0d97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed48d3555d7147f08a8a07ac9adf0d97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42709 } }
I20260811 22:30:37.941449 4476 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed48d3555d7147f08a8a07ac9adf0d97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:30:37.941602 4476 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed48d3555d7147f08a8a07ac9adf0d97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:30:37.941774 4476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed48d3555d7147f08a8a07ac9adf0d97 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:30:37.944273 4476 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed48d3555d7147f08a8a07ac9adf0d97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed48d3555d7147f08a8a07ac9adf0d97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42709 } }
I20260811 22:30:37.944727 4476 leader_election.cc:302] T 00000000000000000000000000000000 P ed48d3555d7147f08a8a07ac9adf0d97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed48d3555d7147f08a8a07ac9adf0d97; no voters:
I20260811 22:30:37.945353 4476 leader_election.cc:288] T 00000000000000000000000000000000 P ed48d3555d7147f08a8a07ac9adf0d97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:30:37.945441 4491 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed48d3555d7147f08a8a07ac9adf0d97 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:30:37.947504 4476 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed48d3555d7147f08a8a07ac9adf0d97 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:30:37.947576 4491 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed48d3555d7147f08a8a07ac9adf0d97 [term 1 LEADER]: Becoming Leader. State: Replica: ed48d3555d7147f08a8a07ac9adf0d97, State: Running, Role: LEADER
I20260811 22:30:37.948164 4491 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed48d3555d7147f08a8a07ac9adf0d97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed48d3555d7147f08a8a07ac9adf0d97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42709 } }
I20260811 22:30:37.952643 4493 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed48d3555d7147f08a8a07ac9adf0d97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed48d3555d7147f08a8a07ac9adf0d97. Latest consensus state: current_term: 1 leader_uuid: "ed48d3555d7147f08a8a07ac9adf0d97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed48d3555d7147f08a8a07ac9adf0d97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42709 } } }
I20260811 22:30:37.953059 4493 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed48d3555d7147f08a8a07ac9adf0d97 [sys.catalog]: This master's current role is: LEADER
I20260811 22:30:37.953547 4492 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed48d3555d7147f08a8a07ac9adf0d97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed48d3555d7147f08a8a07ac9adf0d97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed48d3555d7147f08a8a07ac9adf0d97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42709 } } }
I20260811 22:30:37.953842 4492 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed48d3555d7147f08a8a07ac9adf0d97 [sys.catalog]: This master's current role is: LEADER
I20260811 22:30:37.954144 4508 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:30:37.958297 4479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:37.942680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47931 (local address 127.0.0.1:42709)
0811 22:30:37.942825 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:37.942829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:37.943318 (+ 489us) server_negotiation.cc:407] Connection header received
0811 22:30:37.943458 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:37.943463 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:37.943530 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:37.943604 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:37.944987 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:37.945975 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:37.947286 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:37.947923 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:37.948028 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:37.950566 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:37.950571 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:37.950574 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:30:37.950601 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:37.954292 (+ 3691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:37.955104 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:37.955109 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:37.955112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:37.955248 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:37.955722 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:37.955727 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:37.955729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:37.956347 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:37.956608 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:37.957713 (+ 1105us) server_negotiation.cc:299] Negotiation successful
0811 22:30:37.957931 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260811 22:30:37.966689 4508 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:30:37.975232 4508 catalog_manager.cc:1347] Generated new cluster ID: 99452ba01f114299a5ebfc3c3308fbd7
I20260811 22:30:37.975378 4508 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:30:38.007157 4508 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:30:38.008457 4508 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:30:38.015326 4508 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed48d3555d7147f08a8a07ac9adf0d97: Generated new TSK 0
I20260811 22:30:38.016073 4508 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:30:38.029196 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 4355
I20260811 22:30:38.030194 4465 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:30:38.120620 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 4355
2026-08-11T22:30:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2211 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786487105757681-30120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:30:38 dist-test-slave-gjrz krb5kdc[4533](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:30:38 dist-test-slave-gjrz krb5kdc[4533](info): set up 2 sockets
Aug 11 22:30:38 dist-test-slave-gjrz krb5kdc[4533](info): commencing operation
krb5kdc: starting...
W20260811 22:30:39.398360 30120 mini_kdc.cc:121] Time spent starting KDC: real 1.236s user 0.000s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Aug 11 22:30:39 dist-test-slave-gjrz krb5kdc[4533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:30:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:30:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786487105757681-30120-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786487105757681-30120-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 22:30:39.679729 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38991
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786487105757681-30120-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:30:40.007714 4557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:30:40.008002 4557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:30:40.008145 4557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:30:40.015285 4557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:30:40.015396 4557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:30:40.015475 4557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:30:40.015556 4557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:30:40.015618 4557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:30:40.025127 4557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786487105757681-30120-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:30:40.026316 4557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:30:40.027590 4557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:30:40.038949 4567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:40.039019 4566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:40.038990 4571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:30:40.039641 4557 server_base.cc:1034] running on GCE node
I20260811 22:30:40.040383 4557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:30:40.041767 4557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:30:40.042997 4557 hybrid_clock.cc:648] HybridClock initialized: now 1786487440042952 us; error 30 us; skew 500 ppm
I20260811 22:30:40.043373 4557 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:30:40 dist-test-slave-gjrz krb5kdc[4533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:30:40.046538 4557 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 22:30:40.048035 4557 webserver.cc:460] Webserver started at http://127.0.0.1:41057/ using document root <none> and password file <none>
I20260811 22:30:40.048666 4557 fs_manager.cc:359] Metadata directory not provided
I20260811 22:30:40.048772 4557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:30:40.049095 4557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:30:40.052080 4557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "f9dcf4d1a8684a269093e4b643f6ad37"
format_stamp: "Formatted at 2026-08-11 22:30:40 on dist-test-slave-gjrz"
server_key: "fc6b84c667336d30a54051f6b94f8d25"
server_key_iv: "dc8b48d0c6e409680a249742766784fb"
server_key_version: "encryptionkey@0"
I20260811 22:30:40.052812 4557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "f9dcf4d1a8684a269093e4b643f6ad37"
format_stamp: "Formatted at 2026-08-11 22:30:40 on dist-test-slave-gjrz"
server_key: "fc6b84c667336d30a54051f6b94f8d25"
server_key_iv: "dc8b48d0c6e409680a249742766784fb"
server_key_version: "encryptionkey@0"
I20260811 22:30:40.057794 4557 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260811 22:30:40.061105 4590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:40.062342 4557 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260811 22:30:40.062492 4557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "f9dcf4d1a8684a269093e4b643f6ad37"
format_stamp: "Formatted at 2026-08-11 22:30:40 on dist-test-slave-gjrz"
server_key: "fc6b84c667336d30a54051f6b94f8d25"
server_key_iv: "dc8b48d0c6e409680a249742766784fb"
server_key_version: "encryptionkey@0"
I20260811 22:30:40.062666 4557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:30:40.080192 4557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:30:40.084175 4557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:30:40.084415 4557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:30:40.102085 4557 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38991
I20260811 22:30:40.102109 4697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38991 every 8 connection(s)
I20260811 22:30:40.103353 4557 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:30:40.104813 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 4557
I20260811 22:30:40.104993 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:30:40.105540 30120 external_mini_cluster.cc:1428] Setting key d641aeec4d19471a8f6a7bdc9365a70f
I20260811 22:30:40.107923 4698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:30:40.118345 4702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:40.107202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53689 (local address 127.0.0.1:38991)
0811 22:30:40.107978 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:40.107992 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:40.108049 (+ 57us) server_negotiation.cc:407] Connection header received
0811 22:30:40.109637 (+ 1588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:40.109658 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:40.110016 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:40.110342 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:40.110988 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:40.112332 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:40.114186 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:40.114842 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:40.114972 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:40.116176 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:40.116184 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:40.116197 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:30:40.116725 (+ 528us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:40.116797 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:30:40.116822 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:40.116929 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:40.117170 (+ 241us) server_negotiation.cc:299] Negotiation successful
0811 22:30:40.117436 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":332,"threads_started":1}
I20260811 22:30:40.119665 4698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9dcf4d1a8684a269093e4b643f6ad37: Bootstrap starting.
I20260811 22:30:40.121690 30120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:30:40.122897 4698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9dcf4d1a8684a269093e4b643f6ad37: Neither blocks nor log segments found. Creating new log.
I20260811 22:30:40.124126 4698 log.cc:824] T 00000000000000000000000000000000 P f9dcf4d1a8684a269093e4b643f6ad37: Log is configured to *not* fsync() on all Append() calls
I20260811 22:30:40.126833 4698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9dcf4d1a8684a269093e4b643f6ad37: No bootstrap required, opened a new log
I20260811 22:30:40.133457 4702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:40.126751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56999 (local address 127.0.0.1:38991)
0811 22:30:40.126926 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:40.126933 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:40.127521 (+ 588us) server_negotiation.cc:407] Connection header received
0811 22:30:40.127695 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:40.127699 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:40.127760 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:40.127835 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:40.128429 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:40.129223 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:40.131089 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:40.131615 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:40.131721 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:40.132796 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:40.132801 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:40.132804 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:30:40.132823 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:40.132860 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:30:40.132877 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:40.132947 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:40.133098 (+ 151us) server_negotiation.cc:299] Negotiation successful
0811 22:30:40.133170 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260811 22:30:40.136270 4698 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9dcf4d1a8684a269093e4b643f6ad37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9dcf4d1a8684a269093e4b643f6ad37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38991 } }
I20260811 22:30:40.136624 4698 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9dcf4d1a8684a269093e4b643f6ad37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:30:40.136722 4698 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9dcf4d1a8684a269093e4b643f6ad37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9dcf4d1a8684a269093e4b643f6ad37, State: Initialized, Role: FOLLOWER
I20260811 22:30:40.137323 4698 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9dcf4d1a8684a269093e4b643f6ad37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9dcf4d1a8684a269093e4b643f6ad37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38991 } }
I20260811 22:30:40.137619 4698 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9dcf4d1a8684a269093e4b643f6ad37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:30:40.137758 4698 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9dcf4d1a8684a269093e4b643f6ad37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:30:40.137969 4698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9dcf4d1a8684a269093e4b643f6ad37 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:30:40.140156 4698 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9dcf4d1a8684a269093e4b643f6ad37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9dcf4d1a8684a269093e4b643f6ad37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38991 } }
I20260811 22:30:40.140666 4698 leader_election.cc:302] T 00000000000000000000000000000000 P f9dcf4d1a8684a269093e4b643f6ad37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9dcf4d1a8684a269093e4b643f6ad37; no voters:
I20260811 22:30:40.141304 4698 leader_election.cc:288] T 00000000000000000000000000000000 P f9dcf4d1a8684a269093e4b643f6ad37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:30:40.141443 4718 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9dcf4d1a8684a269093e4b643f6ad37 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:30:40.142132 4718 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9dcf4d1a8684a269093e4b643f6ad37 [term 1 LEADER]: Becoming Leader. State: Replica: f9dcf4d1a8684a269093e4b643f6ad37, State: Running, Role: LEADER
I20260811 22:30:40.142678 4718 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9dcf4d1a8684a269093e4b643f6ad37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9dcf4d1a8684a269093e4b643f6ad37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38991 } }
I20260811 22:30:40.143303 4698 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9dcf4d1a8684a269093e4b643f6ad37 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:30:40.145391 4719 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9dcf4d1a8684a269093e4b643f6ad37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9dcf4d1a8684a269093e4b643f6ad37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9dcf4d1a8684a269093e4b643f6ad37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38991 } } }
I20260811 22:30:40.145761 4719 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9dcf4d1a8684a269093e4b643f6ad37 [sys.catalog]: This master's current role is: LEADER
I20260811 22:30:40.146204 4720 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9dcf4d1a8684a269093e4b643f6ad37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9dcf4d1a8684a269093e4b643f6ad37. Latest consensus state: current_term: 1 leader_uuid: "f9dcf4d1a8684a269093e4b643f6ad37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9dcf4d1a8684a269093e4b643f6ad37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38991 } } }
I20260811 22:30:40.146579 4720 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9dcf4d1a8684a269093e4b643f6ad37 [sys.catalog]: This master's current role is: LEADER
I20260811 22:30:40.148160 4725 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:30:40.152690 4725 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:30:40.161196 4725 catalog_manager.cc:1347] Generated new cluster ID: 01a9c4a899ff443b84651ae1ec3f708c
I20260811 22:30:40.161285 4725 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:30:40.176623 4725 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:30:40.177829 4725 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:30:40.183418 4725 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9dcf4d1a8684a269093e4b643f6ad37: Generated new TSK 0
I20260811 22:30:40.184139 4725 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:30:40.197781 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 4557
W20260811 22:30:40.198830 4689 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53689
W20260811 22:30:40.199545 30120 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 4557: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 22:30:40.199679 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 4557
2026-08-11T22:30:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2080 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786487105757681-30120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:30:40 dist-test-slave-gjrz krb5kdc[4750](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:30:40 dist-test-slave-gjrz krb5kdc[4750](info): set up 2 sockets
Aug 11 22:30:40 dist-test-slave-gjrz krb5kdc[4750](info): commencing operation
krb5kdc: starting...
W20260811 22:30:41.464968 30120 mini_kdc.cc:121] Time spent starting KDC: real 1.222s user 0.004s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Aug 11 22:30:41 dist-test-slave-gjrz krb5kdc[4750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:30:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:30:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786487105757681-30120-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786487105757681-30120-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 22:30:41.744078 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43187
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786487105757681-30120-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:30:42.075282 4775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:30:42.075547 4775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:30:42.075681 4775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:30:42.082762 4775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:30:42.082864 4775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:30:42.082947 4775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:30:42.083029 4775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:30:42.083086 4775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:30:42.092713 4775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786487105757681-30120-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:30:42.093995 4775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:30:42.095286 4775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:30:42.106741 4786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:42.106767 4782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:42.106695 4791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:30:42.107508 4775 server_base.cc:1034] running on GCE node
I20260811 22:30:42.108278 4775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:30:42.109647 4775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:30:42.110865 4775 hybrid_clock.cc:648] HybridClock initialized: now 1786487442110814 us; error 36 us; skew 500 ppm
I20260811 22:30:42.111227 4775 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:30:42 dist-test-slave-gjrz krb5kdc[4750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487442, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:30:42.120606 4775 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 22:30:42.122058 4775 webserver.cc:460] Webserver started at http://127.0.0.1:37073/ using document root <none> and password file <none>
I20260811 22:30:42.122716 4775 fs_manager.cc:359] Metadata directory not provided
I20260811 22:30:42.122848 4775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:30:42.123144 4775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:30:42.126152 4775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "29c511901b6f416a855c05f93c5906c0"
format_stamp: "Formatted at 2026-08-11 22:30:42 on dist-test-slave-gjrz"
server_key: "d726d569724952bf7cbe7d28015f1e8b"
server_key_iv: "116ffbc58f94c1409ee65c9531d0f7b8"
server_key_version: "encryptionkey@0"
I20260811 22:30:42.126933 4775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "29c511901b6f416a855c05f93c5906c0"
format_stamp: "Formatted at 2026-08-11 22:30:42 on dist-test-slave-gjrz"
server_key: "d726d569724952bf7cbe7d28015f1e8b"
server_key_iv: "116ffbc58f94c1409ee65c9531d0f7b8"
server_key_version: "encryptionkey@0"
I20260811 22:30:42.131927 4775 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260811 22:30:42.135203 4804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:42.136454 4775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:30:42.136600 4775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "29c511901b6f416a855c05f93c5906c0"
format_stamp: "Formatted at 2026-08-11 22:30:42 on dist-test-slave-gjrz"
server_key: "d726d569724952bf7cbe7d28015f1e8b"
server_key_iv: "116ffbc58f94c1409ee65c9531d0f7b8"
server_key_version: "encryptionkey@0"
I20260811 22:30:42.136780 4775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:30:42.143502 4775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:30:42.147157 4775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:30:42.147459 4775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:30:42.165323 4775 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43187
I20260811 22:30:42.165369 4912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43187 every 8 connection(s)
I20260811 22:30:42.166601 4775 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:30:42.167893 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 4775
I20260811 22:30:42.168102 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:30:42.168635 30120 external_mini_cluster.cc:1428] Setting key fd0cff4358637895569457022b7534a1
I20260811 22:30:42.171070 4913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:30:42.176466 4923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:42.170352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42593 (local address 127.0.0.1:43187)
0811 22:30:42.171134 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:42.171149 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:42.171200 (+ 51us) server_negotiation.cc:407] Connection header received
0811 22:30:42.172876 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:42.172897 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:42.173217 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:42.173550 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:42.174613 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:42.174619 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:42.174633 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:30:42.174664 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:42.174742 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:30:42.174763 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:42.174956 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:42.175058 (+ 102us) server_negotiation.cc:299] Negotiation successful
0811 22:30:42.175290 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":362,"threads_started":1}
I20260811 22:30:42.180228 30120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:30:42.182984 4913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29c511901b6f416a855c05f93c5906c0: Bootstrap starting.
I20260811 22:30:42.186403 4913 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29c511901b6f416a855c05f93c5906c0: Neither blocks nor log segments found. Creating new log.
I20260811 22:30:42.186585 4923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:42.184350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52877 (local address 127.0.0.1:43187)
0811 22:30:42.184526 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:42.184532 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:42.184920 (+ 388us) server_negotiation.cc:407] Connection header received
0811 22:30:42.185039 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:42.185044 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:42.185099 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:42.185195 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:42.185975 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:42.185980 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:42.185984 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:30:42.186002 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:42.186047 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:30:42.186068 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:42.186182 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:42.186226 (+ 44us) server_negotiation.cc:299] Negotiation successful
0811 22:30:42.186280 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260811 22:30:42.187723 4913 log.cc:824] T 00000000000000000000000000000000 P 29c511901b6f416a855c05f93c5906c0: Log is configured to *not* fsync() on all Append() calls
I20260811 22:30:42.190208 4913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29c511901b6f416a855c05f93c5906c0: No bootstrap required, opened a new log
I20260811 22:30:42.197824 4913 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29c511901b6f416a855c05f93c5906c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c511901b6f416a855c05f93c5906c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43187 } }
I20260811 22:30:42.198182 4913 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29c511901b6f416a855c05f93c5906c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:30:42.198278 4913 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29c511901b6f416a855c05f93c5906c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29c511901b6f416a855c05f93c5906c0, State: Initialized, Role: FOLLOWER
I20260811 22:30:42.198835 4913 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29c511901b6f416a855c05f93c5906c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c511901b6f416a855c05f93c5906c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43187 } }
I20260811 22:30:42.199074 4913 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29c511901b6f416a855c05f93c5906c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:30:42.199199 4913 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29c511901b6f416a855c05f93c5906c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:30:42.199383 4913 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29c511901b6f416a855c05f93c5906c0 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:30:42.201436 4913 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29c511901b6f416a855c05f93c5906c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c511901b6f416a855c05f93c5906c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43187 } }
I20260811 22:30:42.201988 4913 leader_election.cc:302] T 00000000000000000000000000000000 P 29c511901b6f416a855c05f93c5906c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29c511901b6f416a855c05f93c5906c0; no voters:
I20260811 22:30:42.202554 4913 leader_election.cc:288] T 00000000000000000000000000000000 P 29c511901b6f416a855c05f93c5906c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:30:42.202668 4933 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29c511901b6f416a855c05f93c5906c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:30:42.203369 4933 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29c511901b6f416a855c05f93c5906c0 [term 1 LEADER]: Becoming Leader. State: Replica: 29c511901b6f416a855c05f93c5906c0, State: Running, Role: LEADER
I20260811 22:30:42.203933 4933 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29c511901b6f416a855c05f93c5906c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c511901b6f416a855c05f93c5906c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43187 } }
I20260811 22:30:42.204491 4913 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29c511901b6f416a855c05f93c5906c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:30:42.206349 4936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29c511901b6f416a855c05f93c5906c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29c511901b6f416a855c05f93c5906c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c511901b6f416a855c05f93c5906c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43187 } } }
I20260811 22:30:42.206692 4936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29c511901b6f416a855c05f93c5906c0 [sys.catalog]: This master's current role is: LEADER
I20260811 22:30:42.208142 4944 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:30:42.208561 4938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29c511901b6f416a855c05f93c5906c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29c511901b6f416a855c05f93c5906c0. Latest consensus state: current_term: 1 leader_uuid: "29c511901b6f416a855c05f93c5906c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c511901b6f416a855c05f93c5906c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43187 } } }
I20260811 22:30:42.208832 4938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29c511901b6f416a855c05f93c5906c0 [sys.catalog]: This master's current role is: LEADER
I20260811 22:30:42.213126 4944 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:30:42.220898 4944 catalog_manager.cc:1347] Generated new cluster ID: a4cda02d13d242f98044caba98605dae
I20260811 22:30:42.221001 4944 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:30:42.245857 4944 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:30:42.246225 4944 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:30:42.253175 4944 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29c511901b6f416a855c05f93c5906c0: Generated new TSK 0
I20260811 22:30:42.253924 4944 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:30:42.262094 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 4775
W20260811 22:30:42.263670 4908 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42593
W20260811 22:30:42.264329 30120 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 4775: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 22:30:42.264477 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 4775
2026-08-11T22:30:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2066 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786487105757681-30120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:30:42 dist-test-slave-gjrz krb5kdc[4968](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:30:42 dist-test-slave-gjrz krb5kdc[4968](info): set up 2 sockets
Aug 11 22:30:42 dist-test-slave-gjrz krb5kdc[4968](info): commencing operation
krb5kdc: starting...
W20260811 22:30:43.541924 30120 mini_kdc.cc:121] Time spent starting KDC: real 1.232s user 0.000s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Aug 11 22:30:43 dist-test-slave-gjrz krb5kdc[4968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:30:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:30:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786487105757681-30120-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786487105757681-30120-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 22:30:43.818559 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41909
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786487105757681-30120-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:30:44.145816 4994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:30:44.146106 4994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:30:44.146246 4994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:30:44.153306 4994 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:30:44.153419 4994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:30:44.153497 4994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:30:44.153584 4994 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:30:44.153640 4994 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:30:44.163050 4994 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786487105757681-30120-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:30:44.164234 4994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:30:44.165484 4994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:30:44.176824 5002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:44.176787 5003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:44.177227 5005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:30:44.177655 4994 server_base.cc:1034] running on GCE node
I20260811 22:30:44.178350 4994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:30:44.179652 4994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:30:44.180871 4994 hybrid_clock.cc:648] HybridClock initialized: now 1786487444180823 us; error 38 us; skew 500 ppm
I20260811 22:30:44.181238 4994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:30:44 dist-test-slave-gjrz krb5kdc[4968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:30:44.184285 4994 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 22:30:44.185715 4994 webserver.cc:460] Webserver started at http://127.0.0.1:33113/ using document root <none> and password file <none>
I20260811 22:30:44.186347 4994 fs_manager.cc:359] Metadata directory not provided
I20260811 22:30:44.186466 4994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:30:44.186760 4994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:30:44.189761 4994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "ce5768152c9944c0a1298c04936f4138"
format_stamp: "Formatted at 2026-08-11 22:30:44 on dist-test-slave-gjrz"
server_key: "58807e6ec3d3eeda14365ccaf7d87bf5"
server_key_iv: "76b8f8a52fc2e4725826a3bff1bd0c27"
server_key_version: "encryptionkey@0"
I20260811 22:30:44.190735 4994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "ce5768152c9944c0a1298c04936f4138"
format_stamp: "Formatted at 2026-08-11 22:30:44 on dist-test-slave-gjrz"
server_key: "58807e6ec3d3eeda14365ccaf7d87bf5"
server_key_iv: "76b8f8a52fc2e4725826a3bff1bd0c27"
server_key_version: "encryptionkey@0"
I20260811 22:30:44.195586 4994 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260811 22:30:44.198997 5017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:44.200304 4994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:30:44.200456 4994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "ce5768152c9944c0a1298c04936f4138"
format_stamp: "Formatted at 2026-08-11 22:30:44 on dist-test-slave-gjrz"
server_key: "58807e6ec3d3eeda14365ccaf7d87bf5"
server_key_iv: "76b8f8a52fc2e4725826a3bff1bd0c27"
server_key_version: "encryptionkey@0"
I20260811 22:30:44.200623 4994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:30:44.212787 4994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:30:44.216406 4994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:30:44.216648 4994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:30:44.233942 4994 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41909
I20260811 22:30:44.233950 5118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41909 every 8 connection(s)
I20260811 22:30:44.235169 4994 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:30:44.238633 5119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:30:44.244149 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 4994
I20260811 22:30:44.244310 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:30:44.244727 30120 external_mini_cluster.cc:1428] Setting key 72aa5444e9f9c4f03e1c76e0ddf251df
I20260811 22:30:44.249011 5119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce5768152c9944c0a1298c04936f4138: Bootstrap starting.
I20260811 22:30:44.252490 5119 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce5768152c9944c0a1298c04936f4138: Neither blocks nor log segments found. Creating new log.
I20260811 22:30:44.253623 5119 log.cc:824] T 00000000000000000000000000000000 P ce5768152c9944c0a1298c04936f4138: Log is configured to *not* fsync() on all Append() calls
I20260811 22:30:44.256259 5119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce5768152c9944c0a1298c04936f4138: No bootstrap required, opened a new log
Aug 11 22:30:44 dist-test-slave-gjrz krb5kdc[4968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260811 22:30:44.263688 5119 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce5768152c9944c0a1298c04936f4138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce5768152c9944c0a1298c04936f4138" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41909 } }
I20260811 22:30:44.264040 5119 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce5768152c9944c0a1298c04936f4138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:30:44.264147 5119 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce5768152c9944c0a1298c04936f4138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce5768152c9944c0a1298c04936f4138, State: Initialized, Role: FOLLOWER
I20260811 22:30:44.264707 5119 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce5768152c9944c0a1298c04936f4138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce5768152c9944c0a1298c04936f4138" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41909 } }
I20260811 22:30:44.264950 5119 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce5768152c9944c0a1298c04936f4138 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:30:44.265103 5119 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce5768152c9944c0a1298c04936f4138 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:30:44.265290 5119 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce5768152c9944c0a1298c04936f4138 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:30:44.267057 5124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:44.246470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:49111 (local address 127.0.0.1:41909)
0811 22:30:44.247304 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:44.247318 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:44.247360 (+ 42us) server_negotiation.cc:407] Connection header received
0811 22:30:44.249173 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:44.249196 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:44.249566 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:44.249863 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:44.251673 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:44.252974 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:44.254644 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:44.255335 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:44.255504 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:44.259416 (+ 3912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:44.259424 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:44.259438 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:30:44.259476 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:44.262591 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:44.263397 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:44.263404 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:44.263409 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:44.263576 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:44.264035 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:44.264039 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:44.264041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:44.264595 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:44.264952 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:44.265403 (+ 451us) server_negotiation.cc:299] Negotiation successful
0811 22:30:44.265751 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":374,"threads_started":1}
I20260811 22:30:44.268539 5119 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce5768152c9944c0a1298c04936f4138 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce5768152c9944c0a1298c04936f4138" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41909 } }
I20260811 22:30:44.269013 5119 leader_election.cc:302] T 00000000000000000000000000000000 P ce5768152c9944c0a1298c04936f4138 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce5768152c9944c0a1298c04936f4138; no voters:
I20260811 22:30:44.269634 5119 leader_election.cc:288] T 00000000000000000000000000000000 P ce5768152c9944c0a1298c04936f4138 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:30:44.269753 5126 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce5768152c9944c0a1298c04936f4138 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:30:44.270927 5126 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce5768152c9944c0a1298c04936f4138 [term 1 LEADER]: Becoming Leader. State: Replica: ce5768152c9944c0a1298c04936f4138, State: Running, Role: LEADER
I20260811 22:30:44.271060 30120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:30:44.271492 5126 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce5768152c9944c0a1298c04936f4138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce5768152c9944c0a1298c04936f4138" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41909 } }
I20260811 22:30:44.271739 5119 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce5768152c9944c0a1298c04936f4138 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:30:44.274104 5127 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce5768152c9944c0a1298c04936f4138 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce5768152c9944c0a1298c04936f4138" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce5768152c9944c0a1298c04936f4138" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41909 } } }
I20260811 22:30:44.274447 5127 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce5768152c9944c0a1298c04936f4138 [sys.catalog]: This master's current role is: LEADER
I20260811 22:30:44.274915 5133 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce5768152c9944c0a1298c04936f4138 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce5768152c9944c0a1298c04936f4138. Latest consensus state: current_term: 1 leader_uuid: "ce5768152c9944c0a1298c04936f4138" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce5768152c9944c0a1298c04936f4138" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41909 } } }
I20260811 22:30:44.275420 5133 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce5768152c9944c0a1298c04936f4138 [sys.catalog]: This master's current role is: LEADER
I20260811 22:30:44.280095 5137 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:30:44.285604 5137 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:30:44.308669 5124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:44.288514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:53291 (local address 127.0.0.1:41909)
0811 22:30:44.288695 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:44.288702 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:44.290588 (+ 1886us) server_negotiation.cc:407] Connection header received
0811 22:30:44.290763 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:44.290768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:44.290847 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:44.290954 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:44.293012 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:44.294101 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:44.296104 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:44.296892 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:44.296998 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:44.300727 (+ 3729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:44.300735 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:44.300740 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:30:44.300785 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:44.304873 (+ 4088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:44.305958 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:44.305965 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:44.305969 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:44.306135 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:44.306717 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:44.306722 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:44.306725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:44.307164 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:44.307428 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:44.308127 (+ 699us) server_negotiation.cc:299] Negotiation successful
0811 22:30:44.308315 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260811 22:30:44.308866 5137 catalog_manager.cc:1347] Generated new cluster ID: f6ca5e4844414dd39cd3f6497fd14fb6
I20260811 22:30:44.308969 5137 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:30:44.324590 5137 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:30:44.326350 5137 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:30:44.340672 5137 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce5768152c9944c0a1298c04936f4138: Generated new TSK 0
I20260811 22:30:44.341722 5137 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:30:44.346971 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 4994
I20260811 22:30:44.348595 5110 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:30:44.437525 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 4994
W20260811 22:30:44.461165 4965 connection.cc:535] client connection to 127.0.0.1:41909 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41909): Connection reset by peer (error 104)
2026-08-11T22:30:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2169 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786487105757681-30120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:30:44 dist-test-slave-gjrz krb5kdc[5165](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:30:44 dist-test-slave-gjrz krb5kdc[5165](info): set up 2 sockets
Aug 11 22:30:44 dist-test-slave-gjrz krb5kdc[5165](info): commencing operation
krb5kdc: starting...
W20260811 22:30:45.726598 30120 mini_kdc.cc:121] Time spent starting KDC: real 1.247s user 0.001s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Aug 11 22:30:45 dist-test-slave-gjrz krb5kdc[5165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:30:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:30:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786487105757681-30120-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786487105757681-30120-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 22:30:46.012600 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37165
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786487105757681-30120-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:30:46.343291 5184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:30:46.343568 5184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:30:46.343711 5184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:30:46.350708 5184 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:30:46.350817 5184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:30:46.350898 5184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:30:46.350983 5184 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:30:46.351052 5184 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:30:46.360453 5184 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786487105757681-30120-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:30:46.361640 5184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:30:46.362900 5184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:30:46.373880 5200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:46.373950 5198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:46.373973 5195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:30:46.374719 5184 server_base.cc:1034] running on GCE node
I20260811 22:30:46.375480 5184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:30:46.376724 5184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:30:46.377923 5184 hybrid_clock.cc:648] HybridClock initialized: now 1786487446377880 us; error 34 us; skew 500 ppm
I20260811 22:30:46.378248 5184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:30:46 dist-test-slave-gjrz krb5kdc[5165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487446, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:30:46.381227 5184 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 22:30:46.382598 5184 webserver.cc:460] Webserver started at http://127.0.0.1:43665/ using document root <none> and password file <none>
I20260811 22:30:46.383174 5184 fs_manager.cc:359] Metadata directory not provided
I20260811 22:30:46.383302 5184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:30:46.383560 5184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:30:46.386312 5184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "dd57e201b6934fbaaa3a1068de26ea07"
format_stamp: "Formatted at 2026-08-11 22:30:46 on dist-test-slave-gjrz"
server_key: "8e7ace23e8c72e7546c8e35dacf94f8f"
server_key_iv: "28395e42a25900f2797bb65bb4401f40"
server_key_version: "encryptionkey@0"
I20260811 22:30:46.387027 5184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "dd57e201b6934fbaaa3a1068de26ea07"
format_stamp: "Formatted at 2026-08-11 22:30:46 on dist-test-slave-gjrz"
server_key: "8e7ace23e8c72e7546c8e35dacf94f8f"
server_key_iv: "28395e42a25900f2797bb65bb4401f40"
server_key_version: "encryptionkey@0"
I20260811 22:30:46.391816 5184 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260811 22:30:46.394934 5219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:46.396169 5184 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:30:46.396317 5184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "dd57e201b6934fbaaa3a1068de26ea07"
format_stamp: "Formatted at 2026-08-11 22:30:46 on dist-test-slave-gjrz"
server_key: "8e7ace23e8c72e7546c8e35dacf94f8f"
server_key_iv: "28395e42a25900f2797bb65bb4401f40"
server_key_version: "encryptionkey@0"
I20260811 22:30:46.396474 5184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:30:46.409533 5184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:30:46.413146 5184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:30:46.413383 5184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:30:46.430575 5184 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37165
I20260811 22:30:46.430647 5312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37165 every 8 connection(s)
I20260811 22:30:46.431845 5184 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:30:46.435330 5314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:30:46.440917 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 5184
I20260811 22:30:46.441114 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:30:46.441640 30120 external_mini_cluster.cc:1428] Setting key a450e409c2ed045f6ce2c97786d365a5
I20260811 22:30:46.445906 5314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd57e201b6934fbaaa3a1068de26ea07: Bootstrap starting.
I20260811 22:30:46.449295 5314 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd57e201b6934fbaaa3a1068de26ea07: Neither blocks nor log segments found. Creating new log.
I20260811 22:30:46.450520 5314 log.cc:824] T 00000000000000000000000000000000 P dd57e201b6934fbaaa3a1068de26ea07: Log is configured to *not* fsync() on all Append() calls
I20260811 22:30:46.453365 5314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd57e201b6934fbaaa3a1068de26ea07: No bootstrap required, opened a new log
I20260811 22:30:46.454967 5319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:46.443379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:50941 (local address 127.0.0.1:37165)
0811 22:30:46.444160 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:46.444174 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:46.444214 (+ 40us) server_negotiation.cc:407] Connection header received
0811 22:30:46.445824 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:46.445845 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:46.446229 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:46.446527 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:46.447114 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:46.448491 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:46.450405 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:46.451528 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:46.451718 (+ 190us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:46.452898 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:46.452905 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:46.452918 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:30:46.453420 (+ 502us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:46.453499 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:30:46.453531 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:46.453671 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:46.453947 (+ 276us) server_negotiation.cc:299] Negotiation successful
0811 22:30:46.454104 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":364,"threads_started":1}
I20260811 22:30:46.458810 30120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:30:46.461494 5314 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd57e201b6934fbaaa3a1068de26ea07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd57e201b6934fbaaa3a1068de26ea07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37165 } }
I20260811 22:30:46.461819 5314 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd57e201b6934fbaaa3a1068de26ea07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:30:46.461926 5314 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd57e201b6934fbaaa3a1068de26ea07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd57e201b6934fbaaa3a1068de26ea07, State: Initialized, Role: FOLLOWER
I20260811 22:30:46.462486 5314 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd57e201b6934fbaaa3a1068de26ea07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd57e201b6934fbaaa3a1068de26ea07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37165 } }
I20260811 22:30:46.462738 5314 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd57e201b6934fbaaa3a1068de26ea07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:30:46.462890 5314 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd57e201b6934fbaaa3a1068de26ea07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:30:46.463096 5314 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd57e201b6934fbaaa3a1068de26ea07 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:30:46.466017 5314 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd57e201b6934fbaaa3a1068de26ea07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd57e201b6934fbaaa3a1068de26ea07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37165 } }
I20260811 22:30:46.466461 5314 leader_election.cc:302] T 00000000000000000000000000000000 P dd57e201b6934fbaaa3a1068de26ea07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd57e201b6934fbaaa3a1068de26ea07; no voters:
I20260811 22:30:46.467061 5314 leader_election.cc:288] T 00000000000000000000000000000000 P dd57e201b6934fbaaa3a1068de26ea07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:30:46.467203 5334 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd57e201b6934fbaaa3a1068de26ea07 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:30:46.468638 5334 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd57e201b6934fbaaa3a1068de26ea07 [term 1 LEADER]: Becoming Leader. State: Replica: dd57e201b6934fbaaa3a1068de26ea07, State: Running, Role: LEADER
I20260811 22:30:46.469101 5314 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd57e201b6934fbaaa3a1068de26ea07 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:30:46.469236 5334 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd57e201b6934fbaaa3a1068de26ea07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd57e201b6934fbaaa3a1068de26ea07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37165 } }
I20260811 22:30:46.469991 5319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:46.463008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:32959 (local address 127.0.0.1:37165)
0811 22:30:46.463184 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:46.463190 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:46.463579 (+ 389us) server_negotiation.cc:407] Connection header received
0811 22:30:46.463721 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:46.463727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:46.463807 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:46.463906 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:46.464314 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:46.465401 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:46.466674 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:46.467379 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:46.467520 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:46.469113 (+ 1593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:46.469119 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:46.469124 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:30:46.469144 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:46.469188 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:30:46.469210 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:46.469336 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:46.469624 (+ 288us) server_negotiation.cc:299] Negotiation successful
0811 22:30:46.469677 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260811 22:30:46.472584 5338 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd57e201b6934fbaaa3a1068de26ea07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd57e201b6934fbaaa3a1068de26ea07. Latest consensus state: current_term: 1 leader_uuid: "dd57e201b6934fbaaa3a1068de26ea07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd57e201b6934fbaaa3a1068de26ea07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37165 } } }
I20260811 22:30:46.472978 5338 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd57e201b6934fbaaa3a1068de26ea07 [sys.catalog]: This master's current role is: LEADER
I20260811 22:30:46.472702 5337 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd57e201b6934fbaaa3a1068de26ea07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd57e201b6934fbaaa3a1068de26ea07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd57e201b6934fbaaa3a1068de26ea07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37165 } } }
I20260811 22:30:46.473203 5337 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd57e201b6934fbaaa3a1068de26ea07 [sys.catalog]: This master's current role is: LEADER
I20260811 22:30:46.476229 5346 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:30:46.480693 5346 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:30:46.488365 5346 catalog_manager.cc:1347] Generated new cluster ID: 0eb682fd7911499d8234ca2431dcc355
I20260811 22:30:46.488505 5346 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:30:46.502763 5346 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:30:46.503969 5346 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:30:46.514384 5346 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd57e201b6934fbaaa3a1068de26ea07: Generated new TSK 0
I20260811 22:30:46.515031 5346 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:30:46.524919 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 5184
W20260811 22:30:46.525930 5305 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.164:50941
W20260811 22:30:46.526633 30120 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 5184: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 22:30:46.526762 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 5184
W20260811 22:30:46.549149 5162 connection.cc:535] client connection to 127.0.0.1:37165 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37165): Connection reset by peer (error 104)
2026-08-11T22:30:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2088 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786487105757681-30120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:30:46 dist-test-slave-gjrz krb5kdc[5375](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:30:46 dist-test-slave-gjrz krb5kdc[5375](info): set up 2 sockets
Aug 11 22:30:46 dist-test-slave-gjrz krb5kdc[5375](info): commencing operation
krb5kdc: starting...
W20260811 22:30:47.805887 30120 mini_kdc.cc:121] Time spent starting KDC: real 1.238s user 0.004s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Aug 11 22:30:47 dist-test-slave-gjrz krb5kdc[5375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:30:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:30:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786487105757681-30120-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786487105757681-30120-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 22:30:48.092120 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37413
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786487105757681-30120-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:30:48.422056 5396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:30:48.422323 5396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:30:48.422447 5396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:30:48.429505 5396 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:30:48.429608 5396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:30:48.429672 5396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:30:48.429769 5396 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:30:48.429822 5396 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:30:48.439361 5396 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37413
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786487105757681-30120-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:30:48.440506 5396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:30:48.441744 5396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:30:48.452950 5413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:48.453042 5406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:48.453068 5407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:30:48.453740 5396 server_base.cc:1034] running on GCE node
I20260811 22:30:48.454430 5396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:30:48.456003 5396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:30:48.457276 5396 hybrid_clock.cc:648] HybridClock initialized: now 1786487448457223 us; error 41 us; skew 500 ppm
I20260811 22:30:48.457647 5396 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:30:48 dist-test-slave-gjrz krb5kdc[5375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487448, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:30:48.463593 5396 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 22:30:48.464915 5396 webserver.cc:460] Webserver started at http://127.0.0.1:39549/ using document root <none> and password file <none>
I20260811 22:30:48.465513 5396 fs_manager.cc:359] Metadata directory not provided
I20260811 22:30:48.465623 5396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:30:48.465857 5396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:30:48.468732 5396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "0fb14b159ef444c28453a968c6f2b0d7"
format_stamp: "Formatted at 2026-08-11 22:30:48 on dist-test-slave-gjrz"
server_key: "9c10118735ba1bb05b073f549862096f"
server_key_iv: "a15f39d9e79e3d3ad4c5a0321d13614f"
server_key_version: "encryptionkey@0"
I20260811 22:30:48.469420 5396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "0fb14b159ef444c28453a968c6f2b0d7"
format_stamp: "Formatted at 2026-08-11 22:30:48 on dist-test-slave-gjrz"
server_key: "9c10118735ba1bb05b073f549862096f"
server_key_iv: "a15f39d9e79e3d3ad4c5a0321d13614f"
server_key_version: "encryptionkey@0"
I20260811 22:30:48.474156 5396 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260811 22:30:48.477408 5424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:48.478587 5396 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 22:30:48.478731 5396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "0fb14b159ef444c28453a968c6f2b0d7"
format_stamp: "Formatted at 2026-08-11 22:30:48 on dist-test-slave-gjrz"
server_key: "9c10118735ba1bb05b073f549862096f"
server_key_iv: "a15f39d9e79e3d3ad4c5a0321d13614f"
server_key_version: "encryptionkey@0"
I20260811 22:30:48.478910 5396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:30:48.484287 5396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:30:48.488168 5396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:30:48.488431 5396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:30:48.505756 5396 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37413
I20260811 22:30:48.505825 5520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37413 every 8 connection(s)
I20260811 22:30:48.506947 5396 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:30:48.507418 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 5396
I20260811 22:30:48.507612 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:30:48.508163 30120 external_mini_cluster.cc:1428] Setting key b63a3bad1f90319a712d157eb2482345
I20260811 22:30:48.510991 5522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:30:48.515153 5527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:48.509767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:58131 (local address 127.0.0.1:37413)
0811 22:30:48.510493 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:48.510504 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:48.510550 (+ 46us) server_negotiation.cc:407] Connection header received
0811 22:30:48.512214 (+ 1664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:48.512236 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:48.512550 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:48.512835 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:48.513785 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:48.513790 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:48.513802 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:30:48.513816 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:48.513893 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:30:48.513911 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:48.514072 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:48.514164 (+ 92us) server_negotiation.cc:299] Negotiation successful
0811 22:30:48.514317 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":321,"threads_started":1}
I20260811 22:30:48.518623 30120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:30:48.522172 5522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fb14b159ef444c28453a968c6f2b0d7: Bootstrap starting.
I20260811 22:30:48.525115 5527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:48.522644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:51127 (local address 127.0.0.1:37413)
0811 22:30:48.522794 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:48.522799 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:48.523414 (+ 615us) server_negotiation.cc:407] Connection header received
0811 22:30:48.523562 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:48.523567 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:48.523634 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:48.523731 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:48.524540 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:48.524545 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:48.524549 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:30:48.524566 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:48.524607 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:30:48.524626 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:48.524746 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:48.524790 (+ 44us) server_negotiation.cc:299] Negotiation successful
0811 22:30:48.524844 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 22:30:48.525641 5522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fb14b159ef444c28453a968c6f2b0d7: Neither blocks nor log segments found. Creating new log.
I20260811 22:30:48.526717 5522 log.cc:824] T 00000000000000000000000000000000 P 0fb14b159ef444c28453a968c6f2b0d7: Log is configured to *not* fsync() on all Append() calls
I20260811 22:30:48.529451 5522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fb14b159ef444c28453a968c6f2b0d7: No bootstrap required, opened a new log
I20260811 22:30:48.536563 5522 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fb14b159ef444c28453a968c6f2b0d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fb14b159ef444c28453a968c6f2b0d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37413 } }
I20260811 22:30:48.536891 5522 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0fb14b159ef444c28453a968c6f2b0d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:30:48.536983 5522 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fb14b159ef444c28453a968c6f2b0d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fb14b159ef444c28453a968c6f2b0d7, State: Initialized, Role: FOLLOWER
I20260811 22:30:48.537487 5522 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fb14b159ef444c28453a968c6f2b0d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fb14b159ef444c28453a968c6f2b0d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37413 } }
I20260811 22:30:48.537696 5522 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0fb14b159ef444c28453a968c6f2b0d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:30:48.537837 5522 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fb14b159ef444c28453a968c6f2b0d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:30:48.538007 5522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fb14b159ef444c28453a968c6f2b0d7 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:30:48.540155 5522 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fb14b159ef444c28453a968c6f2b0d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fb14b159ef444c28453a968c6f2b0d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37413 } }
I20260811 22:30:48.540623 5522 leader_election.cc:302] T 00000000000000000000000000000000 P 0fb14b159ef444c28453a968c6f2b0d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0fb14b159ef444c28453a968c6f2b0d7; no voters:
I20260811 22:30:48.541180 5522 leader_election.cc:288] T 00000000000000000000000000000000 P 0fb14b159ef444c28453a968c6f2b0d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:30:48.541299 5537 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fb14b159ef444c28453a968c6f2b0d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:30:48.541926 5537 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fb14b159ef444c28453a968c6f2b0d7 [term 1 LEADER]: Becoming Leader. State: Replica: 0fb14b159ef444c28453a968c6f2b0d7, State: Running, Role: LEADER
I20260811 22:30:48.542466 5537 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fb14b159ef444c28453a968c6f2b0d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fb14b159ef444c28453a968c6f2b0d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37413 } }
I20260811 22:30:48.543061 5522 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fb14b159ef444c28453a968c6f2b0d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:30:48.544905 5538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fb14b159ef444c28453a968c6f2b0d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0fb14b159ef444c28453a968c6f2b0d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fb14b159ef444c28453a968c6f2b0d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37413 } } }
I20260811 22:30:48.545622 5538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fb14b159ef444c28453a968c6f2b0d7 [sys.catalog]: This master's current role is: LEADER
I20260811 22:30:48.545285 5539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fb14b159ef444c28453a968c6f2b0d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fb14b159ef444c28453a968c6f2b0d7. Latest consensus state: current_term: 1 leader_uuid: "0fb14b159ef444c28453a968c6f2b0d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fb14b159ef444c28453a968c6f2b0d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37413 } } }
I20260811 22:30:48.546761 5539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fb14b159ef444c28453a968c6f2b0d7 [sys.catalog]: This master's current role is: LEADER
I20260811 22:30:48.551074 5547 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:30:48.554957 5547 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:30:48.562809 5547 catalog_manager.cc:1347] Generated new cluster ID: 9df774d334a4407486e3abcb3dc14904
I20260811 22:30:48.562906 5547 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:30:48.578666 5547 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:30:48.579008 5547 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:30:48.586145 5547 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0fb14b159ef444c28453a968c6f2b0d7: Generated new TSK 0
I20260811 22:30:48.586796 5547 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:30:48.602278 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 5396
W20260811 22:30:48.603488 5514 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.164:58131
W20260811 22:30:48.604130 30120 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 5396: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 22:30:48.604252 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 5396
2026-08-11T22:30:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2073 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786487105757681-30120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:30:48 dist-test-slave-gjrz krb5kdc[5577](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:30:48 dist-test-slave-gjrz krb5kdc[5577](info): set up 2 sockets
Aug 11 22:30:48 dist-test-slave-gjrz krb5kdc[5577](info): commencing operation
krb5kdc: starting...
W20260811 22:30:49.875950 30120 mini_kdc.cc:121] Time spent starting KDC: real 1.235s user 0.004s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Aug 11 22:30:50 dist-test-slave-gjrz krb5kdc[5577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:30:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:30:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786487105757681-30120-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786487105757681-30120-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 22:30:50.155972 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41359
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786487105757681-30120-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41359
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:30:50.483702 5604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:30:50.484050 5604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:30:50.484202 5604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:30:50.491324 5604 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:30:50.491416 5604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:30:50.491467 5604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:30:50.491528 5604 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:30:50.491571 5604 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:30:50.501001 5604 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41359
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786487105757681-30120-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:30:50.502123 5604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:30:50.503321 5604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:30:50.514599 5613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:50.514717 5611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:50.514716 5616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:30:50.515864 5604 server_base.cc:1034] running on GCE node
I20260811 22:30:50.516520 5604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:30:50.517722 5604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:30:50.518918 5604 hybrid_clock.cc:648] HybridClock initialized: now 1786487450518879 us; error 28 us; skew 500 ppm
I20260811 22:30:50.519237 5604 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:30:50 dist-test-slave-gjrz krb5kdc[5577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:30:50.522174 5604 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 22:30:50.523572 5604 webserver.cc:460] Webserver started at http://127.0.0.1:37765/ using document root <none> and password file <none>
I20260811 22:30:50.524158 5604 fs_manager.cc:359] Metadata directory not provided
I20260811 22:30:50.524261 5604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:30:50.524511 5604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:30:50.527253 5604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "9effbf1b5ae14ec8b0f012734b8918d5"
format_stamp: "Formatted at 2026-08-11 22:30:50 on dist-test-slave-gjrz"
server_key: "59f26a00f7d5f45ef4c07205ded4d110"
server_key_iv: "530eb4d5048a755363151fca3c8bf8a5"
server_key_version: "encryptionkey@0"
I20260811 22:30:50.528039 5604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "9effbf1b5ae14ec8b0f012734b8918d5"
format_stamp: "Formatted at 2026-08-11 22:30:50 on dist-test-slave-gjrz"
server_key: "59f26a00f7d5f45ef4c07205ded4d110"
server_key_iv: "530eb4d5048a755363151fca3c8bf8a5"
server_key_version: "encryptionkey@0"
I20260811 22:30:50.532750 5604 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 22:30:50.535902 5627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:50.537067 5604 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:30:50.537206 5604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "9effbf1b5ae14ec8b0f012734b8918d5"
format_stamp: "Formatted at 2026-08-11 22:30:50 on dist-test-slave-gjrz"
server_key: "59f26a00f7d5f45ef4c07205ded4d110"
server_key_iv: "530eb4d5048a755363151fca3c8bf8a5"
server_key_version: "encryptionkey@0"
I20260811 22:30:50.537359 5604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:30:50.562151 5604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:30:50.565690 5604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:30:50.565907 5604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:30:50.583184 5604 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41359
I20260811 22:30:50.583222 5722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41359 every 8 connection(s)
I20260811 22:30:50.584411 5604 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:30:50.587812 5725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:30:50.592123 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 5604
I20260811 22:30:50.592299 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:30:50.592859 30120 external_mini_cluster.cc:1428] Setting key 73d8402addffde74deea582ff4fefb3a
I20260811 22:30:50.598717 5725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9effbf1b5ae14ec8b0f012734b8918d5: Bootstrap starting.
I20260811 22:30:50.602181 5725 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9effbf1b5ae14ec8b0f012734b8918d5: Neither blocks nor log segments found. Creating new log.
I20260811 22:30:50.603331 5725 log.cc:824] T 00000000000000000000000000000000 P 9effbf1b5ae14ec8b0f012734b8918d5: Log is configured to *not* fsync() on all Append() calls
I20260811 22:30:50.606467 5725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9effbf1b5ae14ec8b0f012734b8918d5: No bootstrap required, opened a new log
Aug 11 22:30:50 dist-test-slave-gjrz krb5kdc[5577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260811 22:30:50.614306 5725 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9effbf1b5ae14ec8b0f012734b8918d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9effbf1b5ae14ec8b0f012734b8918d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41359 } }
I20260811 22:30:50.614707 5725 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9effbf1b5ae14ec8b0f012734b8918d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:30:50.614825 5725 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9effbf1b5ae14ec8b0f012734b8918d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9effbf1b5ae14ec8b0f012734b8918d5, State: Initialized, Role: FOLLOWER
I20260811 22:30:50.615402 5725 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9effbf1b5ae14ec8b0f012734b8918d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9effbf1b5ae14ec8b0f012734b8918d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41359 } }
I20260811 22:30:50.615638 5725 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9effbf1b5ae14ec8b0f012734b8918d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:30:50.615643 5728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:50.594482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:51205 (local address 127.0.0.1:41359)
0811 22:30:50.595187 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:50.595197 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:50.595236 (+ 39us) server_negotiation.cc:407] Connection header received
0811 22:30:50.597282 (+ 2046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:50.597303 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:50.597652 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:50.597950 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:50.599784 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:50.601108 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:50.602843 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:50.603567 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:50.603685 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:50.608401 (+ 4716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:50.608411 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:50.608425 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:30:50.608461 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:50.611473 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:50.612240 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:50.612249 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:50.612255 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:50.612416 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:50.612866 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:50.612869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:50.612871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:50.613374 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:50.613673 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:50.614174 (+ 501us) server_negotiation.cc:299] Negotiation successful
0811 22:30:50.614453 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":302,"threads_started":1}
I20260811 22:30:50.615780 5725 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9effbf1b5ae14ec8b0f012734b8918d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:30:50.615994 5725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9effbf1b5ae14ec8b0f012734b8918d5 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:30:50.618279 5725 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9effbf1b5ae14ec8b0f012734b8918d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9effbf1b5ae14ec8b0f012734b8918d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41359 } }
I20260811 22:30:50.618757 5725 leader_election.cc:302] T 00000000000000000000000000000000 P 9effbf1b5ae14ec8b0f012734b8918d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9effbf1b5ae14ec8b0f012734b8918d5; no voters:
I20260811 22:30:50.619290 30120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:30:50.619392 5725 leader_election.cc:288] T 00000000000000000000000000000000 P 9effbf1b5ae14ec8b0f012734b8918d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:30:50.619473 5732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9effbf1b5ae14ec8b0f012734b8918d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:30:50.621141 5725 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9effbf1b5ae14ec8b0f012734b8918d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:30:50.630290 5732 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9effbf1b5ae14ec8b0f012734b8918d5 [term 1 LEADER]: Becoming Leader. State: Replica: 9effbf1b5ae14ec8b0f012734b8918d5, State: Running, Role: LEADER
I20260811 22:30:50.630839 5732 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9effbf1b5ae14ec8b0f012734b8918d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9effbf1b5ae14ec8b0f012734b8918d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41359 } }
I20260811 22:30:50.633709 5733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9effbf1b5ae14ec8b0f012734b8918d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9effbf1b5ae14ec8b0f012734b8918d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9effbf1b5ae14ec8b0f012734b8918d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41359 } } }
I20260811 22:30:50.633771 5734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9effbf1b5ae14ec8b0f012734b8918d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9effbf1b5ae14ec8b0f012734b8918d5. Latest consensus state: current_term: 1 leader_uuid: "9effbf1b5ae14ec8b0f012734b8918d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9effbf1b5ae14ec8b0f012734b8918d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41359 } } }
I20260811 22:30:50.634153 5733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9effbf1b5ae14ec8b0f012734b8918d5 [sys.catalog]: This master's current role is: LEADER
I20260811 22:30:50.634160 5734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9effbf1b5ae14ec8b0f012734b8918d5 [sys.catalog]: This master's current role is: LEADER
I20260811 22:30:50.648247 5728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:50.631238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:36577 (local address 127.0.0.1:41359)
0811 22:30:50.631423 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:50.631430 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:50.631865 (+ 435us) server_negotiation.cc:407] Connection header received
0811 22:30:50.632008 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:50.632013 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:50.632086 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:50.632176 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:50.633954 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:50.635093 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:50.637073 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:50.637773 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:50.637893 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:50.641144 (+ 3251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:50.641152 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:50.641157 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:30:50.641199 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:50.645191 (+ 3992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:50.645893 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:50.645898 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:50.645900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:50.646012 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:50.646445 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:50.646450 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:50.646454 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:50.646888 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:50.647280 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:50.647713 (+ 433us) server_negotiation.cc:299] Negotiation successful
0811 22:30:50.647891 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260811 22:30:50.649322 5767 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9effbf1b5ae14ec8b0f012734b8918d5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 22:30:50.649508 5767 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 22:30:50.649688 5762 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:30:50.651865 5762 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:30:50.660539 5762 catalog_manager.cc:1347] Generated new cluster ID: c79cf0cf82374ce08cbf406630d2f497
I20260811 22:30:50.660636 5762 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:30:50.674991 5762 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:30:50.676149 5762 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:30:50.683015 5762 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9effbf1b5ae14ec8b0f012734b8918d5: Generated new TSK 0
I20260811 22:30:50.683748 5762 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:30:50.698246 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 5604
I20260811 22:30:50.699309 5718 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:30:50.789108 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 5604
W20260811 22:30:50.814796 5575 connection.cc:535] client connection to 127.0.0.1:41359 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41359): Connection reset by peer (error 104)
2026-08-11T22:30:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2192 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786487105757681-30120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:30:50 dist-test-slave-gjrz krb5kdc[5778](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:30:50 dist-test-slave-gjrz krb5kdc[5778](info): set up 2 sockets
Aug 11 22:30:50 dist-test-slave-gjrz krb5kdc[5778](info): commencing operation
krb5kdc: starting...
W20260811 22:30:52.066985 30120 mini_kdc.cc:121] Time spent starting KDC: real 1.233s user 0.001s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Aug 11 22:30:52 dist-test-slave-gjrz krb5kdc[5778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:30:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:30:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786487105757681-30120-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786487105757681-30120-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 22:30:52.362069 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37079
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786487105757681-30120-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:30:52.694294 5812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:30:52.694577 5812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:30:52.694710 5812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:30:52.701753 5812 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:30:52.701864 5812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:30:52.701952 5812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:30:52.702040 5812 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:30:52.702096 5812 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:30:52.711504 5812 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37079
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786487105757681-30120-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:30:52.712706 5812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:30:52.713948 5812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:30:52.724882 5820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:52.724928 5823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:52.724953 5827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:30:52.725953 5812 server_base.cc:1034] running on GCE node
I20260811 22:30:52.726641 5812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:30:52.727957 5812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:30:52.729156 5812 hybrid_clock.cc:648] HybridClock initialized: now 1786487452729112 us; error 36 us; skew 500 ppm
I20260811 22:30:52.729475 5812 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:30:52 dist-test-slave-gjrz krb5kdc[5778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:30:52.732496 5812 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 22:30:52.733886 5812 webserver.cc:460] Webserver started at http://127.0.0.1:40291/ using document root <none> and password file <none>
I20260811 22:30:52.734547 5812 fs_manager.cc:359] Metadata directory not provided
I20260811 22:30:52.734678 5812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:30:52.734951 5812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:30:52.737854 5812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "b13ac74c2e284e0ca6f1208d712d5b90"
format_stamp: "Formatted at 2026-08-11 22:30:52 on dist-test-slave-gjrz"
server_key: "7d50f5ce1c4bbef55d045f2e59bc21a4"
server_key_iv: "84da84e7994c7b521208a7fec47a3ba8"
server_key_version: "encryptionkey@0"
I20260811 22:30:52.738592 5812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "b13ac74c2e284e0ca6f1208d712d5b90"
format_stamp: "Formatted at 2026-08-11 22:30:52 on dist-test-slave-gjrz"
server_key: "7d50f5ce1c4bbef55d045f2e59bc21a4"
server_key_iv: "84da84e7994c7b521208a7fec47a3ba8"
server_key_version: "encryptionkey@0"
I20260811 22:30:52.743464 5812 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260811 22:30:52.746641 5839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:52.747843 5812 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:30:52.747999 5812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "b13ac74c2e284e0ca6f1208d712d5b90"
format_stamp: "Formatted at 2026-08-11 22:30:52 on dist-test-slave-gjrz"
server_key: "7d50f5ce1c4bbef55d045f2e59bc21a4"
server_key_iv: "84da84e7994c7b521208a7fec47a3ba8"
server_key_version: "encryptionkey@0"
I20260811 22:30:52.748170 5812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:30:52.759922 5812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:30:52.763731 5812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:30:52.763968 5812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:30:52.781092 5812 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37079
I20260811 22:30:52.781126 5946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37079 every 8 connection(s)
I20260811 22:30:52.782332 5812 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:30:52.785758 5947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:30:52.786865 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 5812
I20260811 22:30:52.787022 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:30:52.787508 30120 external_mini_cluster.cc:1428] Setting key 577adfe4366194df772e750473960b8e
I20260811 22:30:52.796592 5947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b13ac74c2e284e0ca6f1208d712d5b90: Bootstrap starting.
I20260811 22:30:52.800472 5947 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b13ac74c2e284e0ca6f1208d712d5b90: Neither blocks nor log segments found. Creating new log.
I20260811 22:30:52.801432 5947 log.cc:824] T 00000000000000000000000000000000 P b13ac74c2e284e0ca6f1208d712d5b90: Log is configured to *not* fsync() on all Append() calls
I20260811 22:30:52.804131 5947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b13ac74c2e284e0ca6f1208d712d5b90: No bootstrap required, opened a new log
I20260811 22:30:52.805887 5952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:52.793787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:43111 (local address 127.0.0.1:37079)
0811 22:30:52.794586 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:52.794600 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:52.794650 (+ 50us) server_negotiation.cc:407] Connection header received
0811 22:30:52.796916 (+ 2266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:52.796943 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:52.797418 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:52.797828 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:52.798341 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:52.800311 (+ 1970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:52.801648 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:52.802433 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:52.802616 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:52.803763 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:52.803770 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:52.803784 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:30:52.804308 (+ 524us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:52.804379 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:30:52.804403 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:52.804560 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:52.804836 (+ 276us) server_negotiation.cc:299] Negotiation successful
0811 22:30:52.804970 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":325,"threads_started":1}
I20260811 22:30:52.809463 30120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:30:52.811800 5947 raft_consensus.cc:348] T 00000000000000000000000000000000 P b13ac74c2e284e0ca6f1208d712d5b90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b13ac74c2e284e0ca6f1208d712d5b90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37079 } }
I20260811 22:30:52.812224 5947 raft_consensus.cc:374] T 00000000000000000000000000000000 P b13ac74c2e284e0ca6f1208d712d5b90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:30:52.812335 5947 raft_consensus.cc:729] T 00000000000000000000000000000000 P b13ac74c2e284e0ca6f1208d712d5b90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b13ac74c2e284e0ca6f1208d712d5b90, State: Initialized, Role: FOLLOWER
I20260811 22:30:52.812889 5947 consensus_queue.cc:260] T 00000000000000000000000000000000 P b13ac74c2e284e0ca6f1208d712d5b90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b13ac74c2e284e0ca6f1208d712d5b90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37079 } }
I20260811 22:30:52.813128 5947 raft_consensus.cc:388] T 00000000000000000000000000000000 P b13ac74c2e284e0ca6f1208d712d5b90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:30:52.813254 5947 raft_consensus.cc:482] T 00000000000000000000000000000000 P b13ac74c2e284e0ca6f1208d712d5b90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:30:52.813450 5947 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b13ac74c2e284e0ca6f1208d712d5b90 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:30:52.815595 5947 raft_consensus.cc:504] T 00000000000000000000000000000000 P b13ac74c2e284e0ca6f1208d712d5b90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b13ac74c2e284e0ca6f1208d712d5b90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37079 } }
I20260811 22:30:52.816147 5947 leader_election.cc:302] T 00000000000000000000000000000000 P b13ac74c2e284e0ca6f1208d712d5b90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b13ac74c2e284e0ca6f1208d712d5b90; no voters:
I20260811 22:30:52.816850 5947 leader_election.cc:288] T 00000000000000000000000000000000 P b13ac74c2e284e0ca6f1208d712d5b90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:30:52.816910 5972 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b13ac74c2e284e0ca6f1208d712d5b90 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:30:52.817531 5972 raft_consensus.cc:686] T 00000000000000000000000000000000 P b13ac74c2e284e0ca6f1208d712d5b90 [term 1 LEADER]: Becoming Leader. State: Replica: b13ac74c2e284e0ca6f1208d712d5b90, State: Running, Role: LEADER
I20260811 22:30:52.818069 5972 consensus_queue.cc:237] T 00000000000000000000000000000000 P b13ac74c2e284e0ca6f1208d712d5b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b13ac74c2e284e0ca6f1208d712d5b90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37079 } }
I20260811 22:30:52.819851 5947 sys_catalog.cc:564] T 00000000000000000000000000000000 P b13ac74c2e284e0ca6f1208d712d5b90 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:30:52.820627 5976 sys_catalog.cc:455] T 00000000000000000000000000000000 P b13ac74c2e284e0ca6f1208d712d5b90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b13ac74c2e284e0ca6f1208d712d5b90. Latest consensus state: current_term: 1 leader_uuid: "b13ac74c2e284e0ca6f1208d712d5b90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b13ac74c2e284e0ca6f1208d712d5b90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37079 } } }
I20260811 22:30:52.821027 5976 sys_catalog.cc:458] T 00000000000000000000000000000000 P b13ac74c2e284e0ca6f1208d712d5b90 [sys.catalog]: This master's current role is: LEADER
I20260811 22:30:52.822017 5981 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:30:52.823868 5952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:52.813544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:44751 (local address 127.0.0.1:37079)
0811 22:30:52.813739 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:52.813745 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:52.816239 (+ 2494us) server_negotiation.cc:407] Connection header received
0811 22:30:52.816407 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:52.816411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:52.816467 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:52.816549 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:52.817241 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:52.818388 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:52.820303 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:52.821013 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:52.821144 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:52.822626 (+ 1482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:52.822633 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:52.822637 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:30:52.822656 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:52.822720 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:30:52.822742 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:52.822864 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:52.823478 (+ 614us) server_negotiation.cc:299] Negotiation successful
0811 22:30:52.823532 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 22:30:52.821846 5974 sys_catalog.cc:455] T 00000000000000000000000000000000 P b13ac74c2e284e0ca6f1208d712d5b90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b13ac74c2e284e0ca6f1208d712d5b90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b13ac74c2e284e0ca6f1208d712d5b90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37079 } } }
I20260811 22:30:52.824067 5974 sys_catalog.cc:458] T 00000000000000000000000000000000 P b13ac74c2e284e0ca6f1208d712d5b90 [sys.catalog]: This master's current role is: LEADER
I20260811 22:30:52.834609 5981 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:30:52.842382 5981 catalog_manager.cc:1347] Generated new cluster ID: 77e8ee50b3d4408b86442ecd3e4d61a4
I20260811 22:30:52.842517 5981 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:30:52.870110 5981 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:30:52.871356 5981 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:30:52.882498 5981 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b13ac74c2e284e0ca6f1208d712d5b90: Generated new TSK 0
I20260811 22:30:52.883205 5981 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:30:52.890619 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 5812
W20260811 22:30:52.891633 5940 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.164:43111
W20260811 22:30:52.892277 30120 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 5812: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 22:30:52.892397 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 5812
W20260811 22:30:52.922267 5776 connection.cc:535] client connection to 127.0.0.1:37079 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37079): Connection reset by peer (error 104)
2026-08-11T22:30:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2107 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786487105757681-30120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:30:53 dist-test-slave-gjrz krb5kdc[6008](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:30:53 dist-test-slave-gjrz krb5kdc[6008](info): set up 2 sockets
Aug 11 22:30:53 dist-test-slave-gjrz krb5kdc[6008](info): commencing operation
krb5kdc: starting...
W20260811 22:30:54.174332 30120 mini_kdc.cc:121] Time spent starting KDC: real 1.233s user 0.000s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Aug 11 22:30:54 dist-test-slave-gjrz krb5kdc[6008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:30:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:30:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786487105757681-30120-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786487105757681-30120-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 22:30:54.458081 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45195
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786487105757681-30120-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45195
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:30:54.780665 6029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:30:54.780921 6029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:30:54.781057 6029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:30:54.788017 6029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:30:54.788110 6029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:30:54.788183 6029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:30:54.788273 6029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:30:54.788328 6029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:30:54.797643 6029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45195
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786487105757681-30120-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:30:54.798755 6029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:30:54.799981 6029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:30:54.810963 6045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:54.811004 6048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:54.811033 6046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:30:54.811911 6029 server_base.cc:1034] running on GCE node
I20260811 22:30:54.812635 6029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:30:54.813875 6029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:30:54.815078 6029 hybrid_clock.cc:648] HybridClock initialized: now 1786487454815034 us; error 32 us; skew 500 ppm
I20260811 22:30:54.815457 6029 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:30:54 dist-test-slave-gjrz krb5kdc[6008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:30:54.822831 6029 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 22:30:54.824298 6029 webserver.cc:460] Webserver started at http://127.0.0.1:45039/ using document root <none> and password file <none>
I20260811 22:30:54.824951 6029 fs_manager.cc:359] Metadata directory not provided
I20260811 22:30:54.825055 6029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:30:54.825299 6029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:30:54.828166 6029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "268f6b43875043cf95db551c8d069d9d"
format_stamp: "Formatted at 2026-08-11 22:30:54 on dist-test-slave-gjrz"
server_key: "1b07be9ab34739d14ed68e0ba8f1d207"
server_key_iv: "0e22863ad6b5ae3ab662d0027cd12e29"
server_key_version: "encryptionkey@0"
I20260811 22:30:54.828863 6029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "268f6b43875043cf95db551c8d069d9d"
format_stamp: "Formatted at 2026-08-11 22:30:54 on dist-test-slave-gjrz"
server_key: "1b07be9ab34739d14ed68e0ba8f1d207"
server_key_iv: "0e22863ad6b5ae3ab662d0027cd12e29"
server_key_version: "encryptionkey@0"
I20260811 22:30:54.833720 6029 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260811 22:30:54.836931 6068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:54.838212 6029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260811 22:30:54.838342 6029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "268f6b43875043cf95db551c8d069d9d"
format_stamp: "Formatted at 2026-08-11 22:30:54 on dist-test-slave-gjrz"
server_key: "1b07be9ab34739d14ed68e0ba8f1d207"
server_key_iv: "0e22863ad6b5ae3ab662d0027cd12e29"
server_key_version: "encryptionkey@0"
I20260811 22:30:54.838521 6029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:30:54.844003 6029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:30:54.847934 6029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:30:54.848172 6029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:30:54.865621 6029 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45195
I20260811 22:30:54.865684 6163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45195 every 8 connection(s)
I20260811 22:30:54.866911 6029 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:30:54.870469 6165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:30:54.871225 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 6029
I20260811 22:30:54.871430 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:30:54.871850 30120 external_mini_cluster.cc:1428] Setting key 312d94b0996d13fb64fca42182dbf82d
I20260811 22:30:54.879609 6172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:54.873571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:41911 (local address 127.0.0.1:45195)
0811 22:30:54.874403 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:54.874414 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:54.874463 (+ 49us) server_negotiation.cc:407] Connection header received
0811 22:30:54.876355 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:54.876381 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:54.876705 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:54.877035 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:54.878077 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:54.878084 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:54.878097 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:30:54.878112 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:54.878173 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:30:54.878197 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:54.878372 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:54.878474 (+ 102us) server_negotiation.cc:299] Negotiation successful
0811 22:30:54.878665 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":369,"threads_started":1}
I20260811 22:30:54.883057 6165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 268f6b43875043cf95db551c8d069d9d: Bootstrap starting.
I20260811 22:30:54.883093 30120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:30:54.886834 6165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 268f6b43875043cf95db551c8d069d9d: Neither blocks nor log segments found. Creating new log.
I20260811 22:30:54.888171 6165 log.cc:824] T 00000000000000000000000000000000 P 268f6b43875043cf95db551c8d069d9d: Log is configured to *not* fsync() on all Append() calls
I20260811 22:30:54.890799 6172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:54.888317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:60569 (local address 127.0.0.1:45195)
0811 22:30:54.888449 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:54.888452 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:54.888829 (+ 377us) server_negotiation.cc:407] Connection header received
0811 22:30:54.888989 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:54.888994 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:54.889052 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:54.889160 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:54.890169 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:54.890173 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:54.890176 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:30:54.890189 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:54.890228 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:30:54.890242 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:54.890328 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:54.890458 (+ 130us) server_negotiation.cc:299] Negotiation successful
0811 22:30:54.890513 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260811 22:30:54.891376 6165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 268f6b43875043cf95db551c8d069d9d: No bootstrap required, opened a new log
I20260811 22:30:54.898324 6165 raft_consensus.cc:348] T 00000000000000000000000000000000 P 268f6b43875043cf95db551c8d069d9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268f6b43875043cf95db551c8d069d9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45195 } }
I20260811 22:30:54.898667 6165 raft_consensus.cc:374] T 00000000000000000000000000000000 P 268f6b43875043cf95db551c8d069d9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:30:54.898752 6165 raft_consensus.cc:729] T 00000000000000000000000000000000 P 268f6b43875043cf95db551c8d069d9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 268f6b43875043cf95db551c8d069d9d, State: Initialized, Role: FOLLOWER
I20260811 22:30:54.899245 6165 consensus_queue.cc:260] T 00000000000000000000000000000000 P 268f6b43875043cf95db551c8d069d9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268f6b43875043cf95db551c8d069d9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45195 } }
I20260811 22:30:54.899524 6165 raft_consensus.cc:388] T 00000000000000000000000000000000 P 268f6b43875043cf95db551c8d069d9d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:30:54.899648 6165 raft_consensus.cc:482] T 00000000000000000000000000000000 P 268f6b43875043cf95db551c8d069d9d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:30:54.899792 6165 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 268f6b43875043cf95db551c8d069d9d [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:30:54.901846 6165 raft_consensus.cc:504] T 00000000000000000000000000000000 P 268f6b43875043cf95db551c8d069d9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268f6b43875043cf95db551c8d069d9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45195 } }
I20260811 22:30:54.902383 6165 leader_election.cc:302] T 00000000000000000000000000000000 P 268f6b43875043cf95db551c8d069d9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 268f6b43875043cf95db551c8d069d9d; no voters:
I20260811 22:30:54.902920 6165 leader_election.cc:288] T 00000000000000000000000000000000 P 268f6b43875043cf95db551c8d069d9d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:30:54.903038 6186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 268f6b43875043cf95db551c8d069d9d [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:30:54.903673 6186 raft_consensus.cc:686] T 00000000000000000000000000000000 P 268f6b43875043cf95db551c8d069d9d [term 1 LEADER]: Becoming Leader. State: Replica: 268f6b43875043cf95db551c8d069d9d, State: Running, Role: LEADER
I20260811 22:30:54.904243 6186 consensus_queue.cc:237] T 00000000000000000000000000000000 P 268f6b43875043cf95db551c8d069d9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268f6b43875043cf95db551c8d069d9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45195 } }
I20260811 22:30:54.904784 6165 sys_catalog.cc:564] T 00000000000000000000000000000000 P 268f6b43875043cf95db551c8d069d9d [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:30:54.906781 6187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 268f6b43875043cf95db551c8d069d9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "268f6b43875043cf95db551c8d069d9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268f6b43875043cf95db551c8d069d9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45195 } } }
I20260811 22:30:54.907193 6187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 268f6b43875043cf95db551c8d069d9d [sys.catalog]: This master's current role is: LEADER
I20260811 22:30:54.907954 6188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 268f6b43875043cf95db551c8d069d9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 268f6b43875043cf95db551c8d069d9d. Latest consensus state: current_term: 1 leader_uuid: "268f6b43875043cf95db551c8d069d9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268f6b43875043cf95db551c8d069d9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45195 } } }
I20260811 22:30:54.908331 6188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 268f6b43875043cf95db551c8d069d9d [sys.catalog]: This master's current role is: LEADER
I20260811 22:30:54.909624 6196 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:30:54.914777 6196 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:30:54.923198 6196 catalog_manager.cc:1347] Generated new cluster ID: d13f28213ec4473f980da120e52ed665
I20260811 22:30:54.923311 6196 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:30:54.947475 6196 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:30:54.947861 6196 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:30:54.956727 6196 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 268f6b43875043cf95db551c8d069d9d: Generated new TSK 0
I20260811 22:30:54.957644 6196 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:30:54.972289 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 6029
W20260811 22:30:54.973399 6156 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.164:41911
W20260811 22:30:54.974165 30120 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 6029: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 22:30:54.974290 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 6029
2026-08-11T22:30:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2078 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786487105757681-30120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:30:55 dist-test-slave-gjrz krb5kdc[6213](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:30:55 dist-test-slave-gjrz krb5kdc[6213](info): set up 2 sockets
Aug 11 22:30:55 dist-test-slave-gjrz krb5kdc[6213](info): commencing operation
krb5kdc: starting...
W20260811 22:30:56.246714 30120 mini_kdc.cc:121] Time spent starting KDC: real 1.227s user 0.001s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Aug 11 22:30:56 dist-test-slave-gjrz krb5kdc[6213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:30:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:30:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.106.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
I20260811 22:30:56.518798 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:41555
--webserver_interface=127.29.106.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:34815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:41555
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:30:56.844014 6241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:30:56.844285 6241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:30:56.844424 6241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:30:56.851441 6241 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:30:56.851552 6241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:30:56.851629 6241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:30:56.851711 6241 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:30:56.851768 6241 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:30:56.861034 6241 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:34815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:41555
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:41555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.106.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:30:56.862217 6241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:30:56.863503 6241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:30:56.874987 6253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:56.875012 6268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:56.874943 6263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:30:56.875705 6241 server_base.cc:1034] running on GCE node
I20260811 22:30:56.876444 6241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:30:56.877790 6241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:30:56.879004 6241 hybrid_clock.cc:648] HybridClock initialized: now 1786487456878954 us; error 30 us; skew 500 ppm
I20260811 22:30:56.879405 6241 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:30:56 dist-test-slave-gjrz krb5kdc[6213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487456, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:30:56.885254 6241 init.cc:377] Logged in from keytab as kudu/127.29.106.62@KRBTEST.COM (short username kudu)
I20260811 22:30:56.886634 6241 webserver.cc:460] Webserver started at http://127.29.106.62:41777/ using document root <none> and password file <none>
I20260811 22:30:56.887305 6241 fs_manager.cc:359] Metadata directory not provided
I20260811 22:30:56.887434 6241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:30:56.887717 6241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:30:56.890647 6241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "86c5dbdc6ece4a6f903224d229f17713"
format_stamp: "Formatted at 2026-08-11 22:30:56 on dist-test-slave-gjrz"
server_key: "e4b673b915c17d73c5ad63c446b6fa4c"
server_key_iv: "6a4bc50b2a4ff187c9b781e5c552e6fc"
server_key_version: "encryptionkey@0"
I20260811 22:30:56.891424 6241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "86c5dbdc6ece4a6f903224d229f17713"
format_stamp: "Formatted at 2026-08-11 22:30:56 on dist-test-slave-gjrz"
server_key: "e4b673b915c17d73c5ad63c446b6fa4c"
server_key_iv: "6a4bc50b2a4ff187c9b781e5c552e6fc"
server_key_version: "encryptionkey@0"
I20260811 22:30:56.896209 6241 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260811 22:30:56.899504 6281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:56.900693 6241 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260811 22:30:56.900844 6241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "86c5dbdc6ece4a6f903224d229f17713"
format_stamp: "Formatted at 2026-08-11 22:30:56 on dist-test-slave-gjrz"
server_key: "e4b673b915c17d73c5ad63c446b6fa4c"
server_key_iv: "6a4bc50b2a4ff187c9b781e5c552e6fc"
server_key_version: "encryptionkey@0"
I20260811 22:30:56.901011 6241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:30:56.911880 6241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:30:56.915867 6241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:30:56.916116 6241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:30:56.933656 6241 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.62:41555
I20260811 22:30:56.933691 6381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.62:41555 every 8 connection(s)
I20260811 22:30:56.934906 6241 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:30:56.938647 6382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:30:56.942495 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 6241
I20260811 22:30:56.942680 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:30:56.943099 30120 external_mini_cluster.cc:1428] Setting key ce9c59933feb5759ef8749ee6c9cd066
I20260811 22:30:56.950856 6382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86c5dbdc6ece4a6f903224d229f17713: Bootstrap starting.
I20260811 22:30:56.954583 6382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86c5dbdc6ece4a6f903224d229f17713: Neither blocks nor log segments found. Creating new log.
I20260811 22:30:56.956017 6382 log.cc:824] T 00000000000000000000000000000000 P 86c5dbdc6ece4a6f903224d229f17713: Log is configured to *not* fsync() on all Append() calls
Aug 11 22:30:56 dist-test-slave-gjrz krb5kdc[6213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:30:56.959223 6382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86c5dbdc6ece4a6f903224d229f17713: No bootstrap required, opened a new log
I20260811 22:30:56.968240 6382 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86c5dbdc6ece4a6f903224d229f17713 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86c5dbdc6ece4a6f903224d229f17713" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 41555 } }
I20260811 22:30:56.968708 6382 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86c5dbdc6ece4a6f903224d229f17713 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:30:56.968864 6382 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86c5dbdc6ece4a6f903224d229f17713 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86c5dbdc6ece4a6f903224d229f17713, State: Initialized, Role: FOLLOWER
I20260811 22:30:56.969563 6382 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86c5dbdc6ece4a6f903224d229f17713 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86c5dbdc6ece4a6f903224d229f17713" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 41555 } }
I20260811 22:30:56.969924 6382 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86c5dbdc6ece4a6f903224d229f17713 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:30:56.970062 6382 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86c5dbdc6ece4a6f903224d229f17713 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:30:56.970235 6382 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86c5dbdc6ece4a6f903224d229f17713 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:30:56.973064 6388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:56.944698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56817 (local address 127.29.106.62:41555)
0811 22:30:56.945384 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:56.945396 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:56.945437 (+ 41us) server_negotiation.cc:407] Connection header received
0811 22:30:56.946960 (+ 1523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:56.946980 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:56.947423 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:56.947733 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:56.949404 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:56.950699 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:56.952474 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:56.953133 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:56.953293 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:56.958928 (+ 5635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:56.958939 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:56.958956 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:30:56.958999 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:56.967234 (+ 8235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:56.968347 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:56.968357 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:56.968365 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:56.968568 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:56.969259 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:56.969263 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:56.969266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:56.970019 (+ 753us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:56.970339 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:56.971152 (+ 813us) server_negotiation.cc:299] Negotiation successful
0811 22:30:56.971780 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":300,"threads_started":1}
I20260811 22:30:56.974315 6382 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86c5dbdc6ece4a6f903224d229f17713 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86c5dbdc6ece4a6f903224d229f17713" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 41555 } }
I20260811 22:30:56.974905 6382 leader_election.cc:302] T 00000000000000000000000000000000 P 86c5dbdc6ece4a6f903224d229f17713 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86c5dbdc6ece4a6f903224d229f17713; no voters:
I20260811 22:30:56.977036 30120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:30:56.977195 6382 leader_election.cc:288] T 00000000000000000000000000000000 P 86c5dbdc6ece4a6f903224d229f17713 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:30:56.977316 6390 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86c5dbdc6ece4a6f903224d229f17713 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:30:56.980444 6382 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86c5dbdc6ece4a6f903224d229f17713 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:30:56.982808 6390 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86c5dbdc6ece4a6f903224d229f17713 [term 1 LEADER]: Becoming Leader. State: Replica: 86c5dbdc6ece4a6f903224d229f17713, State: Running, Role: LEADER
I20260811 22:30:56.983304 6390 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86c5dbdc6ece4a6f903224d229f17713 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86c5dbdc6ece4a6f903224d229f17713" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 41555 } }
I20260811 22:30:56.985767 6391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86c5dbdc6ece4a6f903224d229f17713 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86c5dbdc6ece4a6f903224d229f17713" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86c5dbdc6ece4a6f903224d229f17713" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 41555 } } }
I20260811 22:30:56.986162 6391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86c5dbdc6ece4a6f903224d229f17713 [sys.catalog]: This master's current role is: LEADER
I20260811 22:30:56.985808 6395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86c5dbdc6ece4a6f903224d229f17713 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86c5dbdc6ece4a6f903224d229f17713. Latest consensus state: current_term: 1 leader_uuid: "86c5dbdc6ece4a6f903224d229f17713" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86c5dbdc6ece4a6f903224d229f17713" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 41555 } } }
I20260811 22:30:56.990615 6395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86c5dbdc6ece4a6f903224d229f17713 [sys.catalog]: This master's current role is: LEADER
I20260811 22:30:56.991393 6412 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:30:56.996411 6412 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:30:57.010494 6412 catalog_manager.cc:1347] Generated new cluster ID: 043827ebb1db4d56b40d8fb65be23425
I20260811 22:30:57.010664 6412 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:30:57.024529 6412 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:30:57.028062 6412 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:30:57.038301 6412 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86c5dbdc6ece4a6f903224d229f17713: Generated new TSK 0
I20260811 22:30:57.038612 6388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:57.014147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57135 (local address 127.29.106.62:41555)
0811 22:30:57.014332 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:57.014339 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:57.015485 (+ 1146us) server_negotiation.cc:407] Connection header received
0811 22:30:57.016434 (+ 949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:57.016440 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:57.016524 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:57.016611 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:57.020791 (+ 4180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:57.022050 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:57.026117 (+ 4067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:57.027519 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:57.027648 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:57.031121 (+ 3473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:57.031128 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:57.031133 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:30:57.031175 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:57.034784 (+ 3609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:57.035759 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:57.035765 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:57.035768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:57.036021 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:30:57.036622 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:30:57.036627 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:30:57.036629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:30:57.037045 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:57.037317 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:57.038027 (+ 710us) server_negotiation.cc:299] Negotiation successful
0811 22:30:57.038204 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260811 22:30:57.039448 6412 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:30:57.051762 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 6241
I20260811 22:30:57.054792 6377 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:30:57.148134 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 6241
W20260811 22:30:57.176421 6211 connection.cc:535] client connection to 127.29.106.62:41555 recv error: Network error: failed to read from TLS socket (remote: 127.29.106.62:41555): Connection reset by peer (error 104)
2026-08-11T22:30:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2175 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786487105757681-30120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:30:57 dist-test-slave-gjrz krb5kdc[6445](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:30:57 dist-test-slave-gjrz krb5kdc[6445](info): set up 2 sockets
Aug 11 22:30:57 dist-test-slave-gjrz krb5kdc[6445](info): commencing operation
krb5kdc: starting...
W20260811 22:30:58.425295 30120 mini_kdc.cc:121] Time spent starting KDC: real 1.230s user 0.007s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Aug 11 22:30:58 dist-test-slave-gjrz krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:30:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:30:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.106.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
I20260811 22:30:58.712395 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:37545
--webserver_interface=127.29.106.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:43883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:37545
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:30:59.048257 6465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:30:59.048681 6465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:30:59.048856 6465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:30:59.058305 6465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:30:59.058434 6465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:30:59.058521 6465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:30:59.058607 6465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:30:59.058671 6465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:30:59.068887 6465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:43883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:37545
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:37545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.106.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:30:59.070091 6465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:30:59.071426 6465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:30:59.082960 6474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:59.083673 6476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:30:59.083699 6473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:30:59.083889 6465 server_base.cc:1034] running on GCE node
I20260811 22:30:59.084954 6465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:30:59.086688 6465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:30:59.087926 6465 hybrid_clock.cc:648] HybridClock initialized: now 1786487459087860 us; error 45 us; skew 500 ppm
I20260811 22:30:59.088347 6465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:30:59 dist-test-slave-gjrz krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487459, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:30:59.092319 6465 init.cc:377] Logged in from keytab as kudu/127.29.106.62@KRBTEST.COM (short username kudu)
I20260811 22:30:59.094105 6465 webserver.cc:460] Webserver started at http://127.29.106.62:39915/ using document root <none> and password file <none>
I20260811 22:30:59.094987 6465 fs_manager.cc:359] Metadata directory not provided
I20260811 22:30:59.095137 6465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:30:59.095626 6465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:30:59.099783 6465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "7d3fbcb28b764cdb84b26ee3176199d2"
format_stamp: "Formatted at 2026-08-11 22:30:59 on dist-test-slave-gjrz"
server_key: "e10ead6df698ece025149acf951461f6"
server_key_iv: "8121dd77136f593301a4484cd6ad6936"
server_key_version: "encryptionkey@0"
I20260811 22:30:59.100836 6465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "7d3fbcb28b764cdb84b26ee3176199d2"
format_stamp: "Formatted at 2026-08-11 22:30:59 on dist-test-slave-gjrz"
server_key: "e10ead6df698ece025149acf951461f6"
server_key_iv: "8121dd77136f593301a4484cd6ad6936"
server_key_version: "encryptionkey@0"
I20260811 22:30:59.107787 6465 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260811 22:30:59.112300 6495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:30:59.113894 6465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 22:30:59.114081 6465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "7d3fbcb28b764cdb84b26ee3176199d2"
format_stamp: "Formatted at 2026-08-11 22:30:59 on dist-test-slave-gjrz"
server_key: "e10ead6df698ece025149acf951461f6"
server_key_iv: "8121dd77136f593301a4484cd6ad6936"
server_key_version: "encryptionkey@0"
I20260811 22:30:59.114310 6465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:30:59.139045 6465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:30:59.144450 6465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:30:59.144785 6465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:30:59.164955 6465 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.62:37545
I20260811 22:30:59.164969 6600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.62:37545 every 8 connection(s)
I20260811 22:30:59.166265 6465 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:30:59.170065 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 6465
I20260811 22:30:59.170224 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:30:59.170666 30120 external_mini_cluster.cc:1428] Setting key cb248747dcb2c6ca0f3eb0e5bf3e4bdc
I20260811 22:30:59.171231 6601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:30:59.186748 6601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d3fbcb28b764cdb84b26ee3176199d2: Bootstrap starting.
I20260811 22:30:59.191255 6601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d3fbcb28b764cdb84b26ee3176199d2: Neither blocks nor log segments found. Creating new log.
I20260811 22:30:59.192698 6601 log.cc:824] T 00000000000000000000000000000000 P 7d3fbcb28b764cdb84b26ee3176199d2: Log is configured to *not* fsync() on all Append() calls
I20260811 22:30:59.196252 6601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d3fbcb28b764cdb84b26ee3176199d2: No bootstrap required, opened a new log
I20260811 22:30:59.198002 6607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:59.172555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34761 (local address 127.29.106.62:37545)
0811 22:30:59.187422 (+ 14867us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:59.187437 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:59.187501 (+ 64us) server_negotiation.cc:407] Connection header received
0811 22:30:59.189346 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:59.189371 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:59.189755 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:59.190092 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:59.190663 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:59.192088 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:59.193467 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:59.194394 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:59.194554 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:59.195777 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:59.195784 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:59.195798 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:30:59.196332 (+ 534us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:59.196421 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:30:59.196448 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:59.196587 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:59.196868 (+ 281us) server_negotiation.cc:299] Negotiation successful
0811 22:30:59.197064 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14598,"thread_start_us":14362,"threads_started":1}
I20260811 22:30:59.201535 30120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:30:59.207409 6601 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d3fbcb28b764cdb84b26ee3176199d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d3fbcb28b764cdb84b26ee3176199d2" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 37545 } }
I20260811 22:30:59.207969 6601 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d3fbcb28b764cdb84b26ee3176199d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:30:59.208098 6601 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d3fbcb28b764cdb84b26ee3176199d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d3fbcb28b764cdb84b26ee3176199d2, State: Initialized, Role: FOLLOWER
I20260811 22:30:59.208781 6601 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d3fbcb28b764cdb84b26ee3176199d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d3fbcb28b764cdb84b26ee3176199d2" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 37545 } }
I20260811 22:30:59.209067 6601 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d3fbcb28b764cdb84b26ee3176199d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:30:59.209229 6601 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d3fbcb28b764cdb84b26ee3176199d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:30:59.209410 6601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d3fbcb28b764cdb84b26ee3176199d2 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:30:59.212025 6607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:30:59.205713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40205 (local address 127.29.106.62:37545)
0811 22:30:59.205903 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0811 22:30:59.205909 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:30:59.206391 (+ 482us) server_negotiation.cc:407] Connection header received
0811 22:30:59.206577 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:30:59.206581 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:30:59.206635 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:30:59.206710 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:30:59.207186 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:59.208049 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:59.209620 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:30:59.210067 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:30:59.210187 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:30:59.211232 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:30:59.211237 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:30:59.211240 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:30:59.211259 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:30:59.211359 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:30:59.211376 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:30:59.211473 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0811 22:30:59.211670 (+ 197us) server_negotiation.cc:299] Negotiation successful
0811 22:30:59.211722 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260811 22:30:59.212863 6601 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d3fbcb28b764cdb84b26ee3176199d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d3fbcb28b764cdb84b26ee3176199d2" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 37545 } }
I20260811 22:30:59.213462 6601 leader_election.cc:302] T 00000000000000000000000000000000 P 7d3fbcb28b764cdb84b26ee3176199d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d3fbcb28b764cdb84b26ee3176199d2; no voters:
I20260811 22:30:59.214105 6601 leader_election.cc:288] T 00000000000000000000000000000000 P 7d3fbcb28b764cdb84b26ee3176199d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:30:59.214193 6620 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d3fbcb28b764cdb84b26ee3176199d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:30:59.215966 6620 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d3fbcb28b764cdb84b26ee3176199d2 [term 1 LEADER]: Becoming Leader. State: Replica: 7d3fbcb28b764cdb84b26ee3176199d2, State: Running, Role: LEADER
I20260811 22:30:59.216552 6620 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d3fbcb28b764cdb84b26ee3176199d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d3fbcb28b764cdb84b26ee3176199d2" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 37545 } }
I20260811 22:30:59.216753 6601 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d3fbcb28b764cdb84b26ee3176199d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:30:59.219187 6623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d3fbcb28b764cdb84b26ee3176199d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d3fbcb28b764cdb84b26ee3176199d2. Latest consensus state: current_term: 1 leader_uuid: "7d3fbcb28b764cdb84b26ee3176199d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d3fbcb28b764cdb84b26ee3176199d2" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 37545 } } }
I20260811 22:30:59.219775 6623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d3fbcb28b764cdb84b26ee3176199d2 [sys.catalog]: This master's current role is: LEADER
I20260811 22:30:59.219233 6621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d3fbcb28b764cdb84b26ee3176199d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d3fbcb28b764cdb84b26ee3176199d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d3fbcb28b764cdb84b26ee3176199d2" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 37545 } } }
I20260811 22:30:59.220222 6621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d3fbcb28b764cdb84b26ee3176199d2 [sys.catalog]: This master's current role is: LEADER
I20260811 22:30:59.227877 6634 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:30:59.232752 6634 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:30:59.242117 6634 catalog_manager.cc:1347] Generated new cluster ID: 1bfd6f5d65f6462599cc59f696b9baa8
I20260811 22:30:59.242213 6634 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:30:59.254124 6634 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:30:59.255331 6634 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:30:59.277755 6634 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d3fbcb28b764cdb84b26ee3176199d2: Generated new TSK 0
I20260811 22:30:59.278452 6634 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:30:59.286687 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 6465
W20260811 22:30:59.287699 6593 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34761
W20260811 22:30:59.288359 30120 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 6465: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 22:30:59.288487 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 6465
W20260811 22:30:59.317690 6443 connection.cc:535] client connection to 127.29.106.62:37545 recv error: Network error: failed to read from TLS socket (remote: 127.29.106.62:37545): Connection reset by peer (error 104)
2026-08-11T22:30:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2141 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786487105757681-30120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:30:59 dist-test-slave-gjrz krb5kdc[6660](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:30:59 dist-test-slave-gjrz krb5kdc[6660](info): set up 2 sockets
Aug 11 22:30:59 dist-test-slave-gjrz krb5kdc[6660](info): commencing operation
krb5kdc: starting...
W20260811 22:31:00.560640 30120 mini_kdc.cc:121] Time spent starting KDC: real 1.225s user 0.000s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Aug 11 22:31:00 dist-test-slave-gjrz krb5kdc[6660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:31:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:31:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.106.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
I20260811 22:31:00.835798 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:45497
--webserver_interface=127.29.106.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:40661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:45497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:31:01.162781 6679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:31:01.163061 6679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:31:01.163192 6679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:31:01.170322 6679 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:31:01.170425 6679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:31:01.170501 6679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:31:01.170593 6679 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:31:01.170650 6679 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:31:01.180307 6679 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:40661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:45497
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:45497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.106.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:31:01.181478 6679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:31:01.182776 6679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:31:01.194139 6700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:31:01.194178 6697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:31:01.194231 6694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:31:01.194752 6679 server_base.cc:1034] running on GCE node
I20260811 22:31:01.195498 6679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:31:01.196789 6679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:31:01.197990 6679 hybrid_clock.cc:648] HybridClock initialized: now 1786487461197937 us; error 44 us; skew 500 ppm
I20260811 22:31:01.198338 6679 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:31:01 dist-test-slave-gjrz krb5kdc[6660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487461, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:31:01.209331 6679 init.cc:377] Logged in from keytab as kudu/127.29.106.62@KRBTEST.COM (short username kudu)
I20260811 22:31:01.210729 6679 webserver.cc:460] Webserver started at http://127.29.106.62:33905/ using document root <none> and password file <none>
I20260811 22:31:01.211395 6679 fs_manager.cc:359] Metadata directory not provided
I20260811 22:31:01.211503 6679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:31:01.211761 6679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:31:01.214773 6679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "a7d6c695c363488ea6d70e10c8893378"
format_stamp: "Formatted at 2026-08-11 22:31:01 on dist-test-slave-gjrz"
server_key: "3df6e07b88280ba584273ab423a8b517"
server_key_iv: "657b946861beec7ba52895458450a653"
server_key_version: "encryptionkey@0"
I20260811 22:31:01.215638 6679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "a7d6c695c363488ea6d70e10c8893378"
format_stamp: "Formatted at 2026-08-11 22:31:01 on dist-test-slave-gjrz"
server_key: "3df6e07b88280ba584273ab423a8b517"
server_key_iv: "657b946861beec7ba52895458450a653"
server_key_version: "encryptionkey@0"
I20260811 22:31:01.220541 6679 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260811 22:31:01.223937 6718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:31:01.225186 6679 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 22:31:01.225348 6679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "a7d6c695c363488ea6d70e10c8893378"
format_stamp: "Formatted at 2026-08-11 22:31:01 on dist-test-slave-gjrz"
server_key: "3df6e07b88280ba584273ab423a8b517"
server_key_iv: "657b946861beec7ba52895458450a653"
server_key_version: "encryptionkey@0"
I20260811 22:31:01.225533 6679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:31:01.231213 6679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:31:01.235216 6679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:31:01.235504 6679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:31:01.253150 6679 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.62:45497
I20260811 22:31:01.253189 6823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.62:45497 every 8 connection(s)
I20260811 22:31:01.254400 6679 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:31:01.257869 6826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:31:01.260664 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 6679
I20260811 22:31:01.260834 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:31:01.261248 30120 external_mini_cluster.cc:1428] Setting key 17dcca51a202218fae0d109e09829f3d
I20260811 22:31:01.268415 6830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:01.262847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58541 (local address 127.29.106.62:45497)
0811 22:31:01.263673 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:01.263684 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:01.263722 (+ 38us) server_negotiation.cc:407] Connection header received
0811 22:31:01.265346 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:01.265367 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:01.265688 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:01.265998 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:01.266939 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:31:01.266946 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:31:01.266957 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:31:01.266977 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:31:01.267060 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:31:01.267081 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:31:01.267245 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:01.267389 (+ 144us) server_negotiation.cc:299] Negotiation successful
0811 22:31:01.267551 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":368,"threads_started":1}
I20260811 22:31:01.269179 6826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7d6c695c363488ea6d70e10c8893378: Bootstrap starting.
I20260811 22:31:01.271876 30120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:31:01.272359 6826 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7d6c695c363488ea6d70e10c8893378: Neither blocks nor log segments found. Creating new log.
I20260811 22:31:01.273321 6826 log.cc:824] T 00000000000000000000000000000000 P a7d6c695c363488ea6d70e10c8893378: Log is configured to *not* fsync() on all Append() calls
I20260811 22:31:01.276268 6826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7d6c695c363488ea6d70e10c8893378: No bootstrap required, opened a new log
I20260811 22:31:01.278981 6830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:01.276742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44569 (local address 127.29.106.62:45497)
0811 22:31:01.276904 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:01.276910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:01.277301 (+ 391us) server_negotiation.cc:407] Connection header received
0811 22:31:01.277421 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:01.277425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:01.277480 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:01.277583 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:01.278349 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:31:01.278358 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:31:01.278363 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:31:01.278380 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:31:01.278424 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:31:01.278443 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:31:01.278556 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:01.278598 (+ 42us) server_negotiation.cc:299] Negotiation successful
0811 22:31:01.278652 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260811 22:31:01.283849 6826 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7d6c695c363488ea6d70e10c8893378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7d6c695c363488ea6d70e10c8893378" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 45497 } }
I20260811 22:31:01.284173 6826 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7d6c695c363488ea6d70e10c8893378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:31:01.284243 6826 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7d6c695c363488ea6d70e10c8893378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7d6c695c363488ea6d70e10c8893378, State: Initialized, Role: FOLLOWER
I20260811 22:31:01.284739 6826 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7d6c695c363488ea6d70e10c8893378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7d6c695c363488ea6d70e10c8893378" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 45497 } }
I20260811 22:31:01.284945 6826 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7d6c695c363488ea6d70e10c8893378 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:31:01.285054 6826 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7d6c695c363488ea6d70e10c8893378 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:31:01.285168 6826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7d6c695c363488ea6d70e10c8893378 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:31:01.287163 6826 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7d6c695c363488ea6d70e10c8893378 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7d6c695c363488ea6d70e10c8893378" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 45497 } }
I20260811 22:31:01.287637 6826 leader_election.cc:302] T 00000000000000000000000000000000 P a7d6c695c363488ea6d70e10c8893378 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7d6c695c363488ea6d70e10c8893378; no voters:
I20260811 22:31:01.288152 6826 leader_election.cc:288] T 00000000000000000000000000000000 P a7d6c695c363488ea6d70e10c8893378 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:31:01.288252 6838 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7d6c695c363488ea6d70e10c8893378 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:31:01.288935 6838 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7d6c695c363488ea6d70e10c8893378 [term 1 LEADER]: Becoming Leader. State: Replica: a7d6c695c363488ea6d70e10c8893378, State: Running, Role: LEADER
I20260811 22:31:01.289443 6838 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7d6c695c363488ea6d70e10c8893378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7d6c695c363488ea6d70e10c8893378" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 45497 } }
I20260811 22:31:01.289995 6826 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7d6c695c363488ea6d70e10c8893378 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:31:01.292263 6843 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7d6c695c363488ea6d70e10c8893378 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7d6c695c363488ea6d70e10c8893378. Latest consensus state: current_term: 1 leader_uuid: "a7d6c695c363488ea6d70e10c8893378" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7d6c695c363488ea6d70e10c8893378" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 45497 } } }
I20260811 22:31:01.292318 6839 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7d6c695c363488ea6d70e10c8893378 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7d6c695c363488ea6d70e10c8893378" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7d6c695c363488ea6d70e10c8893378" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 45497 } } }
I20260811 22:31:01.292691 6843 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7d6c695c363488ea6d70e10c8893378 [sys.catalog]: This master's current role is: LEADER
I20260811 22:31:01.292695 6839 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7d6c695c363488ea6d70e10c8893378 [sys.catalog]: This master's current role is: LEADER
I20260811 22:31:01.294054 6850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:31:01.298767 6850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:31:01.306805 6850 catalog_manager.cc:1347] Generated new cluster ID: 4ceac95f8eb5437b93350f5e0d52d44d
I20260811 22:31:01.306906 6850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:31:01.339138 6850 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:31:01.339558 6850 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:31:01.347101 6850 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7d6c695c363488ea6d70e10c8893378: Generated new TSK 0
I20260811 22:31:01.347883 6850 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:31:01.363772 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 6679
W20260811 22:31:01.365366 6819 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58541
W20260811 22:31:01.366024 30120 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 6679: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 22:31:01.366147 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 6679
2026-08-11T22:31:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2072 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786487105757681-30120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:31:01 dist-test-slave-gjrz krb5kdc[6874](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:31:01 dist-test-slave-gjrz krb5kdc[6874](info): set up 2 sockets
Aug 11 22:31:01 dist-test-slave-gjrz krb5kdc[6874](info): commencing operation
krb5kdc: starting...
W20260811 22:31:02.644712 30120 mini_kdc.cc:121] Time spent starting KDC: real 1.236s user 0.000s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Aug 11 22:31:02 dist-test-slave-gjrz krb5kdc[6874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:31:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:31:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.106.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
I20260811 22:31:02.937943 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:42893
--webserver_interface=127.29.106.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:33657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:42893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:31:03.266705 6898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:31:03.266978 6898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:31:03.267093 6898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:31:03.274016 6898 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:31:03.274109 6898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:31:03.274179 6898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:31:03.274261 6898 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:31:03.274319 6898 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:31:03.283633 6898 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:33657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:42893
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:42893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.106.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:31:03.284770 6898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:31:03.286027 6898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:31:03.297165 6906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:31:03.297171 6910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:31:03.297097 6907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:31:03.298159 6898 server_base.cc:1034] running on GCE node
I20260811 22:31:03.298898 6898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:31:03.300262 6898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:31:03.301463 6898 hybrid_clock.cc:648] HybridClock initialized: now 1786487463301404 us; error 47 us; skew 500 ppm
I20260811 22:31:03.301796 6898 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:31:03 dist-test-slave-gjrz krb5kdc[6874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487463, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:31:03.304945 6898 init.cc:377] Logged in from keytab as kudu/127.29.106.62@KRBTEST.COM (short username kudu)
I20260811 22:31:03.306449 6898 webserver.cc:460] Webserver started at http://127.29.106.62:34353/ using document root <none> and password file <none>
I20260811 22:31:03.307106 6898 fs_manager.cc:359] Metadata directory not provided
I20260811 22:31:03.307233 6898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:31:03.307552 6898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:31:03.310575 6898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "01094a3f277640d38357d3e48445ca27"
format_stamp: "Formatted at 2026-08-11 22:31:03 on dist-test-slave-gjrz"
server_key: "68bcc384314cea68e0099d9cf75daf47"
server_key_iv: "74e051f923f568222a0091d581f58944"
server_key_version: "encryptionkey@0"
I20260811 22:31:03.311353 6898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "01094a3f277640d38357d3e48445ca27"
format_stamp: "Formatted at 2026-08-11 22:31:03 on dist-test-slave-gjrz"
server_key: "68bcc384314cea68e0099d9cf75daf47"
server_key_iv: "74e051f923f568222a0091d581f58944"
server_key_version: "encryptionkey@0"
I20260811 22:31:03.316339 6898 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260811 22:31:03.319609 6927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:31:03.320819 6898 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260811 22:31:03.320977 6898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "01094a3f277640d38357d3e48445ca27"
format_stamp: "Formatted at 2026-08-11 22:31:03 on dist-test-slave-gjrz"
server_key: "68bcc384314cea68e0099d9cf75daf47"
server_key_iv: "74e051f923f568222a0091d581f58944"
server_key_version: "encryptionkey@0"
I20260811 22:31:03.321152 6898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:31:03.353202 6898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:31:03.357658 6898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:31:03.357918 6898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:31:03.375543 6898 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.62:42893
I20260811 22:31:03.375563 7028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.62:42893 every 8 connection(s)
I20260811 22:31:03.376863 6898 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:31:03.377389 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 6898
I20260811 22:31:03.377558 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:31:03.378144 30120 external_mini_cluster.cc:1428] Setting key 4296e9ae1b66c042ca23b7b6dd77856d
I20260811 22:31:03.381143 7029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 11 22:31:03 dist-test-slave-gjrz krb5kdc[6874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:31:03.391294 7029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01094a3f277640d38357d3e48445ca27: Bootstrap starting.
I20260811 22:31:03.394428 7029 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01094a3f277640d38357d3e48445ca27: Neither blocks nor log segments found. Creating new log.
I20260811 22:31:03.395470 7029 log.cc:824] T 00000000000000000000000000000000 P 01094a3f277640d38357d3e48445ca27: Log is configured to *not* fsync() on all Append() calls
I20260811 22:31:03.398156 7029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01094a3f277640d38357d3e48445ca27: No bootstrap required, opened a new log
I20260811 22:31:03.398685 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:03.379961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37969 (local address 127.29.106.62:42893)
0811 22:31:03.380690 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:03.380702 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:03.380741 (+ 39us) server_negotiation.cc:407] Connection header received
0811 22:31:03.382456 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:03.382477 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:03.382839 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:03.383172 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:03.384610 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:03.385988 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:03.387227 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:03.387917 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:03.388094 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:03.391936 (+ 3842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:31:03.391944 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:31:03.391957 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:31:03.391995 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:31:03.394941 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:03.395618 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:03.395624 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:03.395630 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:03.395790 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:03.396156 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:03.396159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:03.396161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:03.396658 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:31:03.396930 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:03.397367 (+ 437us) server_negotiation.cc:299] Negotiation successful
0811 22:31:03.397743 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":317,"threads_started":1}
I20260811 22:31:03.402351 30120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:31:03.406306 7029 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01094a3f277640d38357d3e48445ca27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01094a3f277640d38357d3e48445ca27" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 42893 } }
I20260811 22:31:03.406740 7029 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01094a3f277640d38357d3e48445ca27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:31:03.406832 7029 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01094a3f277640d38357d3e48445ca27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01094a3f277640d38357d3e48445ca27, State: Initialized, Role: FOLLOWER
I20260811 22:31:03.407486 7029 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01094a3f277640d38357d3e48445ca27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01094a3f277640d38357d3e48445ca27" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 42893 } }
I20260811 22:31:03.407733 7029 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01094a3f277640d38357d3e48445ca27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:31:03.407864 7029 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01094a3f277640d38357d3e48445ca27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:31:03.408016 7029 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01094a3f277640d38357d3e48445ca27 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:31:03.410619 7029 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01094a3f277640d38357d3e48445ca27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01094a3f277640d38357d3e48445ca27" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 42893 } }
I20260811 22:31:03.411096 7029 leader_election.cc:302] T 00000000000000000000000000000000 P 01094a3f277640d38357d3e48445ca27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01094a3f277640d38357d3e48445ca27; no voters:
I20260811 22:31:03.411789 7029 leader_election.cc:288] T 00000000000000000000000000000000 P 01094a3f277640d38357d3e48445ca27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:31:03.411891 7055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01094a3f277640d38357d3e48445ca27 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:31:03.412568 7055 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01094a3f277640d38357d3e48445ca27 [term 1 LEADER]: Becoming Leader. State: Replica: 01094a3f277640d38357d3e48445ca27, State: Running, Role: LEADER
I20260811 22:31:03.413218 7055 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01094a3f277640d38357d3e48445ca27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01094a3f277640d38357d3e48445ca27" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 42893 } }
I20260811 22:31:03.414285 7029 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01094a3f277640d38357d3e48445ca27 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:31:03.415158 7055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01094a3f277640d38357d3e48445ca27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01094a3f277640d38357d3e48445ca27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01094a3f277640d38357d3e48445ca27" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 42893 } } }
I20260811 22:31:03.415499 7055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01094a3f277640d38357d3e48445ca27 [sys.catalog]: This master's current role is: LEADER
I20260811 22:31:03.416234 7056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01094a3f277640d38357d3e48445ca27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01094a3f277640d38357d3e48445ca27. Latest consensus state: current_term: 1 leader_uuid: "01094a3f277640d38357d3e48445ca27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01094a3f277640d38357d3e48445ca27" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 42893 } } }
I20260811 22:31:03.417387 7056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01094a3f277640d38357d3e48445ca27 [sys.catalog]: This master's current role is: LEADER
I20260811 22:31:03.417627 7065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:31:03.423182 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:03.406459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45831 (local address 127.29.106.62:42893)
0811 22:31:03.406644 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:03.406649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:03.406980 (+ 331us) server_negotiation.cc:407] Connection header received
0811 22:31:03.407139 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:03.407145 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:03.407221 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:03.407380 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:03.408826 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:03.409919 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:03.411126 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:03.411890 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:03.412016 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:03.414936 (+ 2920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:31:03.414943 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:31:03.414947 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:31:03.414983 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:31:03.419040 (+ 4057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:03.419898 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:03.419905 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:03.419909 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:03.420084 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:03.420521 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:03.420527 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:03.420530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:03.420926 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:31:03.421201 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:03.422485 (+ 1284us) server_negotiation.cc:299] Negotiation successful
0811 22:31:03.422742 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 22:31:03.430073 7065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:31:03.438097 7065 catalog_manager.cc:1347] Generated new cluster ID: e6db0b3a8df2452eb3f6052ea988b2df
I20260811 22:31:03.438196 7065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:31:03.465284 7065 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:31:03.466777 7065 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:31:03.476676 7065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01094a3f277640d38357d3e48445ca27: Generated new TSK 0
I20260811 22:31:03.477422 7065 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:31:03.494735 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 6898
I20260811 22:31:03.496806 7020 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:31:03.590415 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 6898
2026-08-11T22:31:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2220 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786487105757681-30120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:31:03 dist-test-slave-gjrz krb5kdc[7097](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:31:03 dist-test-slave-gjrz krb5kdc[7097](info): set up 2 sockets
Aug 11 22:31:03 dist-test-slave-gjrz krb5kdc[7097](info): commencing operation
krb5kdc: starting...
W20260811 22:31:04.861131 30120 mini_kdc.cc:121] Time spent starting KDC: real 1.231s user 0.001s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Aug 11 22:31:05 dist-test-slave-gjrz krb5kdc[7097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:31:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:31:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.106.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
I20260811 22:31:05.162618 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:38373
--webserver_interface=127.29.106.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:35765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:38373
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:31:05.499393 7123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:31:05.499729 7123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:31:05.499884 7123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:31:05.507043 7123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:31:05.507141 7123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:31:05.507211 7123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:31:05.507319 7123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:31:05.507383 7123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:31:05.518148 7123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:35765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:38373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:38373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.106.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:31:05.519451 7123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:31:05.520783 7123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:31:05.532197 7131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:31:05.532383 7133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:31:05.532413 7137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:31:05.533151 7123 server_base.cc:1034] running on GCE node
I20260811 22:31:05.533872 7123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:31:05.535190 7123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:31:05.536434 7123 hybrid_clock.cc:648] HybridClock initialized: now 1786487465536397 us; error 31 us; skew 500 ppm
I20260811 22:31:05.536767 7123 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:31:05 dist-test-slave-gjrz krb5kdc[7097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487465, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:31:05.539839 7123 init.cc:377] Logged in from keytab as kudu/127.29.106.62@KRBTEST.COM (short username kudu)
I20260811 22:31:05.541258 7123 webserver.cc:460] Webserver started at http://127.29.106.62:41591/ using document root <none> and password file <none>
I20260811 22:31:05.541889 7123 fs_manager.cc:359] Metadata directory not provided
I20260811 22:31:05.541991 7123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:31:05.542244 7123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:31:05.545199 7123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "27287525b7434fdaa57f8f59c15af88b"
format_stamp: "Formatted at 2026-08-11 22:31:05 on dist-test-slave-gjrz"
server_key: "673463d5ec53b6b01591272485312622"
server_key_iv: "2539a6f1525614e3654cb46e3b268ece"
server_key_version: "encryptionkey@0"
I20260811 22:31:05.545920 7123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "27287525b7434fdaa57f8f59c15af88b"
format_stamp: "Formatted at 2026-08-11 22:31:05 on dist-test-slave-gjrz"
server_key: "673463d5ec53b6b01591272485312622"
server_key_iv: "2539a6f1525614e3654cb46e3b268ece"
server_key_version: "encryptionkey@0"
I20260811 22:31:05.550860 7123 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260811 22:31:05.554262 7150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:31:05.555567 7123 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260811 22:31:05.555728 7123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "27287525b7434fdaa57f8f59c15af88b"
format_stamp: "Formatted at 2026-08-11 22:31:05 on dist-test-slave-gjrz"
server_key: "673463d5ec53b6b01591272485312622"
server_key_iv: "2539a6f1525614e3654cb46e3b268ece"
server_key_version: "encryptionkey@0"
I20260811 22:31:05.555910 7123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:31:05.571381 7123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:31:05.575166 7123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:31:05.575448 7123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:31:05.593305 7123 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.62:38373
I20260811 22:31:05.593339 7248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.62:38373 every 8 connection(s)
I20260811 22:31:05.594519 7123 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:31:05.598070 7249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:31:05.601825 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 7123
I20260811 22:31:05.602031 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:31:05.602607 30120 external_mini_cluster.cc:1428] Setting key 4d1e49ffc6799c9a3fbb0d0eaf1b0c08
I20260811 22:31:05.610040 7249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27287525b7434fdaa57f8f59c15af88b: Bootstrap starting.
I20260811 22:31:05.613013 7249 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27287525b7434fdaa57f8f59c15af88b: Neither blocks nor log segments found. Creating new log.
I20260811 22:31:05.614099 7249 log.cc:824] T 00000000000000000000000000000000 P 27287525b7434fdaa57f8f59c15af88b: Log is configured to *not* fsync() on all Append() calls
I20260811 22:31:05.614946 7254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:05.604324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33961 (local address 127.29.106.62:38373)
0811 22:31:05.605065 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:05.605076 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:05.605115 (+ 39us) server_negotiation.cc:407] Connection header received
0811 22:31:05.606682 (+ 1567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:05.606703 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:05.607083 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:05.607458 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:05.607994 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:05.609307 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:05.610707 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:05.611434 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:05.611611 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:05.612713 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:31:05.612721 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:31:05.612735 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:31:05.613289 (+ 554us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:31:05.613367 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:31:05.613406 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:31:05.613506 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:05.613747 (+ 241us) server_negotiation.cc:299] Negotiation successful
0811 22:31:05.613985 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":348,"threads_started":1}
I20260811 22:31:05.617070 7249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27287525b7434fdaa57f8f59c15af88b: No bootstrap required, opened a new log
I20260811 22:31:05.618672 30120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:31:05.627029 7249 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27287525b7434fdaa57f8f59c15af88b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27287525b7434fdaa57f8f59c15af88b" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 38373 } }
I20260811 22:31:05.627489 7249 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27287525b7434fdaa57f8f59c15af88b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:31:05.627601 7249 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27287525b7434fdaa57f8f59c15af88b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27287525b7434fdaa57f8f59c15af88b, State: Initialized, Role: FOLLOWER
I20260811 22:31:05.628203 7249 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27287525b7434fdaa57f8f59c15af88b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27287525b7434fdaa57f8f59c15af88b" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 38373 } }
I20260811 22:31:05.628450 7249 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27287525b7434fdaa57f8f59c15af88b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:31:05.628597 7249 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27287525b7434fdaa57f8f59c15af88b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:31:05.628800 7249 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27287525b7434fdaa57f8f59c15af88b [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:31:05.630301 7254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:05.623708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54201 (local address 127.29.106.62:38373)
0811 22:31:05.623881 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:05.623888 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:05.624391 (+ 503us) server_negotiation.cc:407] Connection header received
0811 22:31:05.624558 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:05.624561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:05.624618 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:05.624701 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:05.625283 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:05.626005 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:05.627882 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:05.628382 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:05.628503 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:05.629599 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:31:05.629604 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:31:05.629607 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:31:05.629623 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:31:05.629670 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:31:05.629685 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:31:05.629761 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:05.629940 (+ 179us) server_negotiation.cc:299] Negotiation successful
0811 22:31:05.630026 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260811 22:31:05.631659 7249 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27287525b7434fdaa57f8f59c15af88b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27287525b7434fdaa57f8f59c15af88b" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 38373 } }
I20260811 22:31:05.632162 7249 leader_election.cc:302] T 00000000000000000000000000000000 P 27287525b7434fdaa57f8f59c15af88b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27287525b7434fdaa57f8f59c15af88b; no voters:
I20260811 22:31:05.632686 7249 leader_election.cc:288] T 00000000000000000000000000000000 P 27287525b7434fdaa57f8f59c15af88b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:31:05.632798 7262 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27287525b7434fdaa57f8f59c15af88b [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:31:05.634198 7262 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27287525b7434fdaa57f8f59c15af88b [term 1 LEADER]: Becoming Leader. State: Replica: 27287525b7434fdaa57f8f59c15af88b, State: Running, Role: LEADER
I20260811 22:31:05.634693 7262 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27287525b7434fdaa57f8f59c15af88b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27287525b7434fdaa57f8f59c15af88b" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 38373 } }
I20260811 22:31:05.634783 7249 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27287525b7434fdaa57f8f59c15af88b [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:31:05.637526 7265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27287525b7434fdaa57f8f59c15af88b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27287525b7434fdaa57f8f59c15af88b. Latest consensus state: current_term: 1 leader_uuid: "27287525b7434fdaa57f8f59c15af88b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27287525b7434fdaa57f8f59c15af88b" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 38373 } } }
I20260811 22:31:05.637696 7263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27287525b7434fdaa57f8f59c15af88b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27287525b7434fdaa57f8f59c15af88b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27287525b7434fdaa57f8f59c15af88b" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 38373 } } }
I20260811 22:31:05.637988 7263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27287525b7434fdaa57f8f59c15af88b [sys.catalog]: This master's current role is: LEADER
I20260811 22:31:05.637964 7265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27287525b7434fdaa57f8f59c15af88b [sys.catalog]: This master's current role is: LEADER
I20260811 22:31:05.639731 7276 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:31:05.645350 7276 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:31:05.654619 7276 catalog_manager.cc:1347] Generated new cluster ID: aa0f766ec0cf4fa182cc7f403a32bfeb
I20260811 22:31:05.654727 7276 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:31:05.664809 7276 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:31:05.666535 7276 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:31:05.676744 7276 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27287525b7434fdaa57f8f59c15af88b: Generated new TSK 0
I20260811 22:31:05.677489 7276 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:31:05.693814 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 7123
W20260811 22:31:05.694828 7238 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33961
W20260811 22:31:05.695451 30120 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 7123: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 22:31:05.695606 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 7123
2026-08-11T22:31:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2107 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786487105757681-30120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:31:05 dist-test-slave-gjrz krb5kdc[7301](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:31:05 dist-test-slave-gjrz krb5kdc[7301](info): set up 2 sockets
Aug 11 22:31:05 dist-test-slave-gjrz krb5kdc[7301](info): commencing operation
krb5kdc: starting...
W20260811 22:31:06.967132 30120 mini_kdc.cc:121] Time spent starting KDC: real 1.230s user 0.000s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Aug 11 22:31:07 dist-test-slave-gjrz krb5kdc[7301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:31:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:31:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.106.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
I20260811 22:31:07.290365 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:42497
--webserver_interface=127.29.106.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:46687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:42497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:31:07.622877 7328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:31:07.623167 7328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:31:07.623327 7328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:31:07.630497 7328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:31:07.630616 7328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:31:07.630712 7328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:31:07.630793 7328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:31:07.630856 7328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:31:07.640504 7328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:46687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:42497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:42497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.106.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:31:07.641764 7328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:31:07.643038 7328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:31:07.654170 7342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:31:07.654138 7344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:31:07.654757 7339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:31:07.654964 7328 server_base.cc:1034] running on GCE node
I20260811 22:31:07.655758 7328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:31:07.657047 7328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:31:07.658259 7328 hybrid_clock.cc:648] HybridClock initialized: now 1786487467658205 us; error 32 us; skew 500 ppm
I20260811 22:31:07.658591 7328 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:31:07 dist-test-slave-gjrz krb5kdc[7301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487467, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:31:07.661645 7328 init.cc:377] Logged in from keytab as kudu/127.29.106.62@KRBTEST.COM (short username kudu)
I20260811 22:31:07.663034 7328 webserver.cc:460] Webserver started at http://127.29.106.62:44541/ using document root <none> and password file <none>
I20260811 22:31:07.663720 7328 fs_manager.cc:359] Metadata directory not provided
I20260811 22:31:07.663852 7328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:31:07.664194 7328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:31:07.667016 7328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "c10563777302463bafedbee562b3c979"
format_stamp: "Formatted at 2026-08-11 22:31:07 on dist-test-slave-gjrz"
server_key: "20c459e36250479a993339d5ddca0da9"
server_key_iv: "84745f108b85a2b4999dc0841b003d0b"
server_key_version: "encryptionkey@0"
I20260811 22:31:07.667759 7328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "c10563777302463bafedbee562b3c979"
format_stamp: "Formatted at 2026-08-11 22:31:07 on dist-test-slave-gjrz"
server_key: "20c459e36250479a993339d5ddca0da9"
server_key_iv: "84745f108b85a2b4999dc0841b003d0b"
server_key_version: "encryptionkey@0"
I20260811 22:31:07.672632 7328 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260811 22:31:07.676141 7356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:31:07.677376 7328 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 22:31:07.677539 7328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "c10563777302463bafedbee562b3c979"
format_stamp: "Formatted at 2026-08-11 22:31:07 on dist-test-slave-gjrz"
server_key: "20c459e36250479a993339d5ddca0da9"
server_key_iv: "84745f108b85a2b4999dc0841b003d0b"
server_key_version: "encryptionkey@0"
I20260811 22:31:07.677711 7328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:31:07.683254 7328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:31:07.686942 7328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:31:07.687224 7328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:31:07.704768 7328 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.62:42497
I20260811 22:31:07.704802 7450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.62:42497 every 8 connection(s)
I20260811 22:31:07.706053 7328 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:31:07.707531 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 7328
I20260811 22:31:07.707691 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:31:07.708118 30120 external_mini_cluster.cc:1428] Setting key 0aee73c9487a6db0b31913fff7e02783
I20260811 22:31:07.709981 7451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:31:07.716507 7456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:07.709824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42249 (local address 127.29.106.62:42497)
0811 22:31:07.710700 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:07.710714 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:07.710760 (+ 46us) server_negotiation.cc:407] Connection header received
0811 22:31:07.712852 (+ 2092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:07.712878 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:07.713310 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:07.713725 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:07.714654 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:31:07.714662 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:31:07.714677 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:31:07.714695 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:31:07.714794 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:31:07.714827 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:31:07.715027 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:07.715157 (+ 130us) server_negotiation.cc:299] Negotiation successful
0811 22:31:07.715371 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":373,"threads_started":1}
I20260811 22:31:07.720110 30120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:31:07.724161 7451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c10563777302463bafedbee562b3c979: Bootstrap starting.
I20260811 22:31:07.727540 7456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:07.725412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40437 (local address 127.29.106.62:42497)
0811 22:31:07.725566 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:07.725571 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:07.725901 (+ 330us) server_negotiation.cc:407] Connection header received
0811 22:31:07.726021 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:07.726026 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:07.726080 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:07.726186 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:07.726978 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:31:07.726983 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:31:07.726987 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:31:07.727003 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:31:07.727045 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:31:07.727064 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:31:07.727172 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:07.727216 (+ 44us) server_negotiation.cc:299] Negotiation successful
0811 22:31:07.727287 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260811 22:31:07.727612 7451 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c10563777302463bafedbee562b3c979: Neither blocks nor log segments found. Creating new log.
I20260811 22:31:07.728838 7451 log.cc:824] T 00000000000000000000000000000000 P c10563777302463bafedbee562b3c979: Log is configured to *not* fsync() on all Append() calls
I20260811 22:31:07.731376 7451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c10563777302463bafedbee562b3c979: No bootstrap required, opened a new log
I20260811 22:31:07.738770 7451 raft_consensus.cc:348] T 00000000000000000000000000000000 P c10563777302463bafedbee562b3c979 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c10563777302463bafedbee562b3c979" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 42497 } }
I20260811 22:31:07.739066 7451 raft_consensus.cc:374] T 00000000000000000000000000000000 P c10563777302463bafedbee562b3c979 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:31:07.739142 7451 raft_consensus.cc:729] T 00000000000000000000000000000000 P c10563777302463bafedbee562b3c979 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c10563777302463bafedbee562b3c979, State: Initialized, Role: FOLLOWER
I20260811 22:31:07.739657 7451 consensus_queue.cc:260] T 00000000000000000000000000000000 P c10563777302463bafedbee562b3c979 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c10563777302463bafedbee562b3c979" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 42497 } }
I20260811 22:31:07.739848 7451 raft_consensus.cc:388] T 00000000000000000000000000000000 P c10563777302463bafedbee562b3c979 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:31:07.739948 7451 raft_consensus.cc:482] T 00000000000000000000000000000000 P c10563777302463bafedbee562b3c979 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:31:07.740067 7451 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c10563777302463bafedbee562b3c979 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:31:07.742123 7451 raft_consensus.cc:504] T 00000000000000000000000000000000 P c10563777302463bafedbee562b3c979 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c10563777302463bafedbee562b3c979" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 42497 } }
I20260811 22:31:07.742569 7451 leader_election.cc:302] T 00000000000000000000000000000000 P c10563777302463bafedbee562b3c979 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c10563777302463bafedbee562b3c979; no voters:
I20260811 22:31:07.743076 7451 leader_election.cc:288] T 00000000000000000000000000000000 P c10563777302463bafedbee562b3c979 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:31:07.743211 7473 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c10563777302463bafedbee562b3c979 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:31:07.744493 7473 raft_consensus.cc:686] T 00000000000000000000000000000000 P c10563777302463bafedbee562b3c979 [term 1 LEADER]: Becoming Leader. State: Replica: c10563777302463bafedbee562b3c979, State: Running, Role: LEADER
I20260811 22:31:07.745013 7451 sys_catalog.cc:564] T 00000000000000000000000000000000 P c10563777302463bafedbee562b3c979 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:31:07.744947 7473 consensus_queue.cc:237] T 00000000000000000000000000000000 P c10563777302463bafedbee562b3c979 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c10563777302463bafedbee562b3c979" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 42497 } }
I20260811 22:31:07.747609 7478 sys_catalog.cc:455] T 00000000000000000000000000000000 P c10563777302463bafedbee562b3c979 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c10563777302463bafedbee562b3c979. Latest consensus state: current_term: 1 leader_uuid: "c10563777302463bafedbee562b3c979" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c10563777302463bafedbee562b3c979" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 42497 } } }
I20260811 22:31:07.748041 7478 sys_catalog.cc:458] T 00000000000000000000000000000000 P c10563777302463bafedbee562b3c979 [sys.catalog]: This master's current role is: LEADER
I20260811 22:31:07.747655 7475 sys_catalog.cc:455] T 00000000000000000000000000000000 P c10563777302463bafedbee562b3c979 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c10563777302463bafedbee562b3c979" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c10563777302463bafedbee562b3c979" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 42497 } } }
I20260811 22:31:07.748657 7475 sys_catalog.cc:458] T 00000000000000000000000000000000 P c10563777302463bafedbee562b3c979 [sys.catalog]: This master's current role is: LEADER
I20260811 22:31:07.751742 7495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:31:07.755908 7495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:31:07.764127 7495 catalog_manager.cc:1347] Generated new cluster ID: 26a35397d9814e67aa27d0d3fa10e047
I20260811 22:31:07.764225 7495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:31:07.793517 7495 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:31:07.793902 7495 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:31:07.800614 7495 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c10563777302463bafedbee562b3c979: Generated new TSK 0
I20260811 22:31:07.801255 7495 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:31:07.812137 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 7328
W20260811 22:31:07.813230 7446 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42249
W20260811 22:31:07.814150 30120 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 7328: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260811 22:31:07.814319 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 7328
2026-08-11T22:31:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2126 ms)
[----------] 18 tests from AuthTokenIssuingTest (38261 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:31:07 dist-test-slave-gjrz krb5kdc[7519](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:31:07 dist-test-slave-gjrz krb5kdc[7519](info): set up 2 sockets
Aug 11 22:31:07 dist-test-slave-gjrz krb5kdc[7519](info): commencing operation
krb5kdc: starting...
W20260811 22:31:09.107685 30120 mini_kdc.cc:121] Time spent starting KDC: real 1.244s user 0.001s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Aug 11 22:31:09 dist-test-slave-gjrz krb5kdc[7519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:31:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:31:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.106.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
I20260811 22:31:09.406055 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:36013
--webserver_interface=127.29.106.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:37045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:36013,127.29.106.61:40697,127.29.106.60:39473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:31:09.736251 7554 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260811 22:31:09.736574 7554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:31:09.736681 7554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:31:09.736811 7554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:31:09.743856 7554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:31:09.743932 7554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:31:09.744005 7554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:31:09.744077 7554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:31:09.744164 7554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:31:09.753505 7554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:37045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:36013,127.29.106.61:40697,127.29.106.60:39473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:36013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.106.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:31:09.754637 7554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:31:09.755884 7554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:31:09.767134 7566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:31:09.767177 7575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:31:09.767206 7565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:31:09.768180 7554 server_base.cc:1034] running on GCE node
I20260811 22:31:09.768831 7554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:31:09.770031 7554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:31:09.771294 7554 hybrid_clock.cc:648] HybridClock initialized: now 1786487469771185 us; error 36 us; skew 500 ppm
I20260811 22:31:09.771625 7554 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:31:09 dist-test-slave-gjrz krb5kdc[7519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487469, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:31:09.774600 7554 init.cc:377] Logged in from keytab as kudu/127.29.106.62@KRBTEST.COM (short username kudu)
I20260811 22:31:09.776070 7554 webserver.cc:460] Webserver started at http://127.29.106.62:33169/ using document root <none> and password file <none>
I20260811 22:31:09.776708 7554 fs_manager.cc:359] Metadata directory not provided
I20260811 22:31:09.776809 7554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:31:09.777107 7554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:31:09.780021 7554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "60ff5f9a5fe246eaae40121a81ca46f8"
format_stamp: "Formatted at 2026-08-11 22:31:09 on dist-test-slave-gjrz"
server_key: "c3d93ec564fe991811d208bb73513e60"
server_key_iv: "591a63d465a5ecba9e1c3ad53b4622d4"
server_key_version: "encryptionkey@0"
I20260811 22:31:09.780803 7554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "60ff5f9a5fe246eaae40121a81ca46f8"
format_stamp: "Formatted at 2026-08-11 22:31:09 on dist-test-slave-gjrz"
server_key: "c3d93ec564fe991811d208bb73513e60"
server_key_iv: "591a63d465a5ecba9e1c3ad53b4622d4"
server_key_version: "encryptionkey@0"
I20260811 22:31:09.785569 7554 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260811 22:31:09.788720 7595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:31:09.789954 7554 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 22:31:09.790105 7554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "60ff5f9a5fe246eaae40121a81ca46f8"
format_stamp: "Formatted at 2026-08-11 22:31:09 on dist-test-slave-gjrz"
server_key: "c3d93ec564fe991811d208bb73513e60"
server_key_iv: "591a63d465a5ecba9e1c3ad53b4622d4"
server_key_version: "encryptionkey@0"
I20260811 22:31:09.790294 7554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:31:09.806483 7554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:31:09.810120 7554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:31:09.810349 7554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:31:09.827926 7554 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.62:36013
I20260811 22:31:09.827965 7694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.62:36013 every 8 connection(s)
I20260811 22:31:09.829129 7554 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:31:09.831329 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 7554
I20260811 22:31:09.831527 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:31:09.832182 30120 external_mini_cluster.cc:1428] Setting key e9f314ef4ed4b3323bf82291597b144a
I20260811 22:31:09.832688 7695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:31:09.838832 7695 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.106.62" port: 36013 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 22:31:09 dist-test-slave-gjrz krb5kdc[7519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487469, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:31:09.859935 7704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:09.843061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38228 (local address 127.29.106.62:36013)
0811 22:31:09.843766 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:09.843774 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:09.843821 (+ 47us) server_negotiation.cc:407] Connection header received
0811 22:31:09.844945 (+ 1124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:09.844950 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:09.845155 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:09.845325 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:09.846668 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:09.847923 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:09.849244 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:09.849772 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:09.849929 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:09.853296 (+ 3367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:31:09.853304 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:31:09.853315 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:31:09.853343 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:31:09.855992 (+ 2649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:09.856830 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:09.856837 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:09.856839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:09.856986 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:09.857388 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:09.857399 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:09.857401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:09.857833 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:31:09.857990 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:09.858628 (+ 638us) server_negotiation.cc:299] Negotiation successful
0811 22:31:09.858910 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":327,"threads_started":1}
I20260811 22:31:09.859974 7700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:09.841860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:36013 (local address 127.0.0.1:38228)
0811 22:31:09.842587 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:09.842624 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0811 22:31:09.844601 (+ 1977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:31:09.845381 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:31:09.845388 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:31:09.846144 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:31:09.846465 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:31:09.846482 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:31:09.848070 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:31:09.848073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:31:09.849106 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:31:09.849117 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:31:09.850007 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:31:09.850014 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:31:09.850186 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:09.850759 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:31:09.850780 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:31:09.853058 (+ 2278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:31:09.856126 (+ 3068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:31:09.856138 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:31:09.856149 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:31:09.856707 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:31:09.857097 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:31:09.857101 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:31:09.857104 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:31:09.857289 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:31:09.858001 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:31:09.858009 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:31:09.858226 (+ 217us) client_negotiation.cc:769] Sending connection context
0811 22:31:09.858489 (+ 263us) client_negotiation.cc:240] Negotiation successful
0811 22:31:09.859112 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"mutex_wait_us":136,"thread_start_us":281,"threads_started":1}
WARNING: no policy specified for kudu/127.29.106.61@KRBTEST.COM; defaulting to no policy
I20260811 22:31:09.863137 7695 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.106.61" port: 40697 } has no permanent_uuid. Determining permanent_uuid...
I20260811 22:31:09.864208 7700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:09.863833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.61:40697 (local address 127.0.0.1:50306)
0811 22:31:09.863986 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:09.864097 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.61:40697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260811 22:31:09.864528 7601 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.106.61:40697: connect: Connection refused (error 111)
I20260811 22:31:09.865983 7700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:09.865631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.61:40697 (local address 127.0.0.1:50314)
0811 22:31:09.865779 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:09.865848 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.61:40697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260811 22:31:09.866201 7695 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.106.61:40697: Network error: Client connection negotiation failed: client connection to 127.29.106.61:40697: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.61.keytab.
Entry for principal kudu/127.29.106.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.61.keytab.
I20260811 22:31:09.902556 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.61:40697
--webserver_interface=127.29.106.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.61.keytab
--principal=kudu/127.29.106.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:37045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:36013,127.29.106.61:40697,127.29.106.60:39473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 22:31:09.915530 7695 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.106.61" port: 40697 } attempt: 1
I20260811 22:31:09.916699 7700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:09.916235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.61:40697 (local address 127.0.0.1:50324)
0811 22:31:09.916434 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:09.916570 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.61:40697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260811 22:31:09.917699 7700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:09.917330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.61:40697 (local address 127.0.0.1:50340)
0811 22:31:09.917491 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:09.917562 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.61:40697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260811 22:31:09.917891 7695 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.106.61:40697: Network error: Client connection negotiation failed: client connection to 127.29.106.61:40697: connect: Connection refused (error 111)
I20260811 22:31:09.986105 7695 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.106.61" port: 40697 } attempt: 2
I20260811 22:31:09.987253 7700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:09.986792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.61:40697 (local address 127.0.0.1:50342)
0811 22:31:09.987018 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:09.987113 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.61:40697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260811 22:31:09.988281 7700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:09.987915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.61:40697 (local address 127.0.0.1:50352)
0811 22:31:09.988085 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:09.988155 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.61:40697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260811 22:31:09.988476 7695 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.106.61:40697: Network error: Client connection negotiation failed: client connection to 127.29.106.61:40697: connect: Connection refused (error 111)
I20260811 22:31:10.079703 7695 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.106.61" port: 40697 } attempt: 3
I20260811 22:31:10.080926 7700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:10.080479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.61:40697 (local address 127.0.0.1:50354)
0811 22:31:10.080693 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:10.080791 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.61:40697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260811 22:31:10.082005 7700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:10.081623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.61:40697 (local address 127.0.0.1:50356)
0811 22:31:10.081779 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:10.081850 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.61:40697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260811 22:31:10.082209 7695 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.106.61:40697: Network error: Client connection negotiation failed: client connection to 127.29.106.61:40697: connect: Connection refused (error 111)
I20260811 22:31:10.225469 7695 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.106.61" port: 40697 } attempt: 4
I20260811 22:31:10.226815 7700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:10.226183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.61:40697 (local address 127.0.0.1:50370)
0811 22:31:10.226551 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:10.226666 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.61:40697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199}
I20260811 22:31:10.227926 7700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:10.227566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.61:40697 (local address 127.0.0.1:50380)
0811 22:31:10.227722 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:10.227803 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.61:40697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260811 22:31:10.228123 7695 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.106.61:40697: Network error: Client connection negotiation failed: client connection to 127.29.106.61:40697: connect: Connection refused (error 111)
W20260811 22:31:10.233866 7711 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260811 22:31:10.234181 7711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:31:10.234277 7711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:31:10.234400 7711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:31:10.241804 7711 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:31:10.241907 7711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:31:10.242008 7711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:31:10.242079 7711 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:31:10.242162 7711 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:31:10.251616 7711 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:37045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:36013,127.29.106.61:40697,127.29.106.60:39473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.61.keytab
--principal=kudu/127.29.106.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.61:40697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.29.106.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:31:10.252784 7711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:31:10.254029 7711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:31:10.265420 7729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:31:10.265415 7725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:31:10.265929 7727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:31:10.266179 7711 server_base.cc:1034] running on GCE node
I20260811 22:31:10.266932 7711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:31:10.268304 7711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:31:10.269536 7711 hybrid_clock.cc:648] HybridClock initialized: now 1786487470269465 us; error 37 us; skew 500 ppm
I20260811 22:31:10.269848 7711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:31:10 dist-test-slave-gjrz krb5kdc[7519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487470, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:31:10.272998 7711 init.cc:377] Logged in from keytab as kudu/127.29.106.61@KRBTEST.COM (short username kudu)
I20260811 22:31:10.274441 7711 webserver.cc:460] Webserver started at http://127.29.106.61:35967/ using document root <none> and password file <none>
I20260811 22:31:10.275099 7711 fs_manager.cc:359] Metadata directory not provided
I20260811 22:31:10.275202 7711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:31:10.275540 7711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:31:10.278400 7711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/minicluster-data/master-1/data/instance:
uuid: "824f0c26bfad41fcaac01dd3ba274b49"
format_stamp: "Formatted at 2026-08-11 22:31:10 on dist-test-slave-gjrz"
server_key: "1e094dc6e9ec07bd8769c8ca5e06c591"
server_key_iv: "f83827aeaeb6bbe71c341847e8d04744"
server_key_version: "encryptionkey@0"
I20260811 22:31:10.279170 7711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/minicluster-data/master-1/wal/instance:
uuid: "824f0c26bfad41fcaac01dd3ba274b49"
format_stamp: "Formatted at 2026-08-11 22:31:10 on dist-test-slave-gjrz"
server_key: "1e094dc6e9ec07bd8769c8ca5e06c591"
server_key_iv: "f83827aeaeb6bbe71c341847e8d04744"
server_key_version: "encryptionkey@0"
I20260811 22:31:10.284067 7711 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260811 22:31:10.287246 7742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:31:10.288669 7711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260811 22:31:10.288827 7711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/minicluster-data/master-1/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/minicluster-data/master-1/wal
uuid: "824f0c26bfad41fcaac01dd3ba274b49"
format_stamp: "Formatted at 2026-08-11 22:31:10 on dist-test-slave-gjrz"
server_key: "1e094dc6e9ec07bd8769c8ca5e06c591"
server_key_iv: "f83827aeaeb6bbe71c341847e8d04744"
server_key_version: "encryptionkey@0"
I20260811 22:31:10.289026 7711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:31:10.305497 7711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:31:10.309243 7711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:31:10.309477 7711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:31:10.326328 7711 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.61:40697
I20260811 22:31:10.326342 7836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.61:40697 every 8 connection(s)
I20260811 22:31:10.327605 7711 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/minicluster-data/master-1/data/info.pb
I20260811 22:31:10.329980 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 7711
I20260811 22:31:10.330173 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/minicluster-data/master-1/wal/instance
I20260811 22:31:10.330726 30120 external_mini_cluster.cc:1428] Setting key 342367ecc3c62d97ad43e2e0742cefbb
I20260811 22:31:10.331696 7837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:31:10.339994 7837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.106.62" port: 36013 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 22:31:10 dist-test-slave-gjrz krb5kdc[7519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487470, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.61@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:31:10.361209 7704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:10.342967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38234 (local address 127.29.106.62:36013)
0811 22:31:10.343185 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:10.343192 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:10.344394 (+ 1202us) server_negotiation.cc:407] Connection header received
0811 22:31:10.346228 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:10.346235 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:10.346320 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:10.346435 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:10.348296 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:10.348976 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:10.351032 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:10.351464 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:10.351572 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:10.355577 (+ 4005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:31:10.355581 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:31:10.355585 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:31:10.355618 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:31:10.358233 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:10.359012 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:10.359016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:10.359018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:10.359138 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:10.359601 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:10.359605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:10.359607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:10.359877 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:31:10.359986 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:10.360794 (+ 808us) server_negotiation.cc:299] Negotiation successful
0811 22:31:10.360973 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260811 22:31:10.362533 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:10.343423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:36013 (local address 127.0.0.1:38234)
0811 22:31:10.344227 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:10.344270 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0811 22:31:10.345951 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:31:10.346655 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:31:10.346666 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:31:10.347401 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:31:10.348116 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:31:10.348140 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:31:10.349145 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:31:10.349150 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:31:10.350897 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:31:10.350912 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:31:10.351714 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:31:10.351725 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:31:10.352031 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:10.352679 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:31:10.352706 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:31:10.355357 (+ 2651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:31:10.358383 (+ 3026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:31:10.358390 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:31:10.358403 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:31:10.358910 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:31:10.359249 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:31:10.359254 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:31:10.359257 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:31:10.359496 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:31:10.360008 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:31:10.360017 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:31:10.360407 (+ 390us) client_negotiation.cc:769] Sending connection context
0811 22:31:10.360708 (+ 301us) client_negotiation.cc:240] Negotiation successful
0811 22:31:10.361199 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":309,"threads_started":1}
I20260811 22:31:10.364284 7837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.106.61" port: 40697 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.29.106.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.60@KRBTEST.COM" created.
Aug 11 22:31:10 dist-test-slave-gjrz krb5kdc[7519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487470, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.61@KRBTEST.COM for kudu/127.29.106.61@KRBTEST.COM
I20260811 22:31:10.377857 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:10.365113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.61:40697 (local address 127.0.0.1:50396)
0811 22:31:10.365293 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:10.365311 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 22:31:10.365415 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:31:10.366267 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:31:10.366271 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:31:10.366619 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:31:10.366757 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:31:10.366765 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:31:10.368472 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:31:10.368478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:31:10.369280 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:31:10.369290 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:31:10.370052 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:31:10.370058 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:31:10.370227 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:10.370722 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:31:10.370737 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:31:10.372686 (+ 1949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:31:10.375509 (+ 2823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:31:10.375514 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:31:10.375518 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:31:10.375908 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:31:10.376307 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:31:10.376311 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:31:10.376314 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:31:10.376462 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:31:10.377106 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:31:10.377114 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:31:10.377240 (+ 126us) client_negotiation.cc:769] Sending connection context
0811 22:31:10.377357 (+ 117us) client_negotiation.cc:240] Negotiation successful
0811 22:31:10.377568 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260811 22:31:10.378140 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:10.365206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50396 (local address 127.29.106.61:40697)
0811 22:31:10.365808 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:10.365814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:10.365839 (+ 25us) server_negotiation.cc:407] Connection header received
0811 22:31:10.365916 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:10.365924 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:10.366128 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:10.366236 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:10.366900 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:10.368337 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:10.369412 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:10.369917 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:10.370032 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:10.372873 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:31:10.372884 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:31:10.372895 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:31:10.372926 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:31:10.375400 (+ 2474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:10.376045 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:10.376059 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:10.376062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:10.376209 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:10.376565 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:10.376569 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:10.376572 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:10.376992 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:31:10.377204 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:10.377520 (+ 316us) server_negotiation.cc:299] Negotiation successful
0811 22:31:10.377752 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":343,"threads_started":1}
I20260811 22:31:10.380599 7837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.106.60" port: 39473 } has no permanent_uuid. Determining permanent_uuid...
I20260811 22:31:10.381685 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:10.381217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:39473 (local address 127.0.0.1:54880)
0811 22:31:10.381395 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:10.381524 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:39473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260811 22:31:10.381956 7743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.106.60:39473: connect: Connection refused (error 111)
I20260811 22:31:10.383499 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:10.383119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:39473 (local address 127.0.0.1:54888)
0811 22:31:10.383300 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:10.383391 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:39473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260811 22:31:10.383769 7837 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.106.60:39473: Network error: Client connection negotiation failed: client connection to 127.29.106.60:39473: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.60.keytab.
Entry for principal kudu/127.29.106.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.60.keytab.
I20260811 22:31:10.402612 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.60:39473
--webserver_interface=127.29.106.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.60.keytab
--principal=kudu/127.29.106.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:37045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:36013,127.29.106.61:40697,127.29.106.60:39473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 22:31:10.433041 7837 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.106.60" port: 39473 } attempt: 1
I20260811 22:31:10.434185 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:10.433726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:39473 (local address 127.0.0.1:54890)
0811 22:31:10.433909 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:10.434012 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:39473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260811 22:31:10.435160 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:10.434804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:39473 (local address 127.0.0.1:54904)
0811 22:31:10.434965 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:10.435041 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:39473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260811 22:31:10.435385 7837 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.106.60:39473: Network error: Client connection negotiation failed: client connection to 127.29.106.60:39473: connect: Connection refused (error 111)
I20260811 22:31:10.503580 7837 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.106.60" port: 39473 } attempt: 2
I20260811 22:31:10.504588 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:10.504164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:39473 (local address 127.0.0.1:54912)
0811 22:31:10.504354 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:10.504440 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:39473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260811 22:31:10.505551 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:10.505231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:39473 (local address 127.0.0.1:54928)
0811 22:31:10.505372 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:10.505438 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:39473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260811 22:31:10.505766 7837 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.106.60:39473: Network error: Client connection negotiation failed: client connection to 127.29.106.60:39473: connect: Connection refused (error 111)
I20260811 22:31:10.527355 7695 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.106.61" port: 40697 } attempt: 5
Aug 11 22:31:10 dist-test-slave-gjrz krb5kdc[7519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487469, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for kudu/127.29.106.61@KRBTEST.COM
I20260811 22:31:10.541568 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:10.528068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50402 (local address 127.29.106.61:40697)
0811 22:31:10.528242 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:10.528248 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:10.528268 (+ 20us) server_negotiation.cc:407] Connection header received
0811 22:31:10.528428 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:10.528434 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:10.528510 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:10.528603 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:10.529660 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:10.530772 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:10.532410 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:10.533072 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:10.533212 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:10.536574 (+ 3362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:31:10.536579 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:31:10.536582 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:31:10.536614 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:31:10.539040 (+ 2426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:10.539859 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:10.539863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:10.539866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:10.539981 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:10.540382 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:10.540385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:10.540387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:10.540647 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:31:10.540753 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:10.541202 (+ 449us) server_negotiation.cc:299] Negotiation successful
0811 22:31:10.541370 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260811 22:31:10.541812 7700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:10.527946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.61:40697 (local address 127.0.0.1:50402)
0811 22:31:10.528146 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:10.528167 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 22:31:10.528294 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:31:10.528680 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:31:10.528685 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:31:10.529217 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:31:10.529474 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:31:10.529488 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:31:10.530951 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:31:10.530956 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:31:10.532223 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:31:10.532239 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:31:10.533229 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:31:10.533233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:31:10.533380 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:10.533946 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:31:10.533963 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:31:10.536389 (+ 2426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:31:10.539193 (+ 2804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:31:10.539198 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:31:10.539202 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:31:10.539757 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:31:10.540095 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:31:10.540101 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:31:10.540104 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:31:10.540278 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:31:10.540765 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:31:10.540770 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:31:10.540942 (+ 172us) client_negotiation.cc:769] Sending connection context
0811 22:31:10.541095 (+ 153us) client_negotiation.cc:240] Negotiation successful
0811 22:31:10.541359 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260811 22:31:10.542958 7695 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.106.60" port: 39473 } has no permanent_uuid. Determining permanent_uuid...
I20260811 22:31:10.544054 7700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:10.543704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:39473 (local address 127.0.0.1:54940)
0811 22:31:10.543852 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:10.543925 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:39473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260811 22:31:10.544950 7700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:10.544674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:39473 (local address 127.0.0.1:54944)
0811 22:31:10.544794 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:10.544858 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:39473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260811 22:31:10.545137 7695 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.106.60:39473: Network error: Client connection negotiation failed: client connection to 127.29.106.60:39473: connect: Connection refused (error 111)
I20260811 22:31:10.596333 7695 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.106.60" port: 39473 } attempt: 1
I20260811 22:31:10.596948 7837 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.106.60" port: 39473 } attempt: 3
I20260811 22:31:10.597942 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:10.597432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:39473 (local address 127.0.0.1:54950)
0811 22:31:10.597627 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:10.597746 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:39473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260811 22:31:10.597944 7700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:10.597039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:39473 (local address 127.0.0.1:54948)
0811 22:31:10.597672 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:10.597762 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:39473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260811 22:31:10.599027 7700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:10.598654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:39473 (local address 127.0.0.1:54962)
0811 22:31:10.598804 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:10.598891 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:39473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260811 22:31:10.599234 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:10.598706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:39473 (local address 127.0.0.1:54978)
0811 22:31:10.598983 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:10.599094 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:39473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170}
W20260811 22:31:10.599458 7695 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.106.60:39473: Network error: Client connection negotiation failed: client connection to 127.29.106.60:39473: connect: Connection refused (error 111)
W20260811 22:31:10.599498 7837 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.106.60:39473: Network error: Client connection negotiation failed: client connection to 127.29.106.60:39473: connect: Connection refused (error 111)
I20260811 22:31:10.667745 7695 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.106.60" port: 39473 } attempt: 2
I20260811 22:31:10.668895 7700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:10.668428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:39473 (local address 127.0.0.1:54986)
0811 22:31:10.668646 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:10.668742 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:39473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260811 22:31:10.669816 7700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:10.669489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:39473 (local address 127.0.0.1:54988)
0811 22:31:10.669638 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:10.669711 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:39473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260811 22:31:10.670012 7695 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.106.60:39473: Network error: Client connection negotiation failed: client connection to 127.29.106.60:39473: connect: Connection refused (error 111)
W20260811 22:31:10.731354 7852 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260811 22:31:10.731675 7852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:31:10.731772 7852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:31:10.731889 7852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:31:10.738795 7852 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:31:10.738880 7852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:31:10.738966 7852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:31:10.739032 7852 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:31:10.739118 7852 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:31:10.742717 7837 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.106.60" port: 39473 } attempt: 4
I20260811 22:31:10.743888 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:10.743307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:39473 (local address 127.0.0.1:54992)
0811 22:31:10.743573 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:10.743684 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:39473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260811 22:31:10.744917 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:10.744535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:39473 (local address 127.0.0.1:55002)
0811 22:31:10.744694 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:10.744785 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:39473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260811 22:31:10.745137 7837 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.106.60:39473: Network error: Client connection negotiation failed: client connection to 127.29.106.60:39473: connect: Connection refused (error 111)
I20260811 22:31:10.749328 7852 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:37045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:36013,127.29.106.61:40697,127.29.106.60:39473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.60.keytab
--principal=kudu/127.29.106.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.60:39473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.29.106.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:31:10.750443 7852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:31:10.751703 7852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:31:10.762759 7861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:31:10.762794 7864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:31:10.763150 7860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:31:10.763602 7852 server_base.cc:1034] running on GCE node
I20260811 22:31:10.764271 7852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:31:10.765479 7852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:31:10.766690 7852 hybrid_clock.cc:648] HybridClock initialized: now 1786487470766642 us; error 25 us; skew 500 ppm
I20260811 22:31:10.767036 7852 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:31:10 dist-test-slave-gjrz krb5kdc[7519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487470, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:31:10.770046 7852 init.cc:377] Logged in from keytab as kudu/127.29.106.60@KRBTEST.COM (short username kudu)
I20260811 22:31:10.771409 7852 webserver.cc:460] Webserver started at http://127.29.106.60:41593/ using document root <none> and password file <none>
I20260811 22:31:10.772032 7852 fs_manager.cc:359] Metadata directory not provided
I20260811 22:31:10.772135 7852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:31:10.772430 7852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:31:10.775190 7852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/minicluster-data/master-2/data/instance:
uuid: "ac5daa92df8c465e83cc56ca6df19bd7"
format_stamp: "Formatted at 2026-08-11 22:31:10 on dist-test-slave-gjrz"
server_key: "6435bc985b0ddae30c71874d0bc0720d"
server_key_iv: "0ee2ad15de93cd3635df34d9e02cadae"
server_key_version: "encryptionkey@0"
I20260811 22:31:10.775985 7852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/minicluster-data/master-2/wal/instance:
uuid: "ac5daa92df8c465e83cc56ca6df19bd7"
format_stamp: "Formatted at 2026-08-11 22:31:10 on dist-test-slave-gjrz"
server_key: "6435bc985b0ddae30c71874d0bc0720d"
server_key_iv: "0ee2ad15de93cd3635df34d9e02cadae"
server_key_version: "encryptionkey@0"
I20260811 22:31:10.776206 7695 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.106.60" port: 39473 } attempt: 3
I20260811 22:31:10.777346 7700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:10.776762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:39473 (local address 127.0.0.1:55012)
0811 22:31:10.777022 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:10.777135 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:39473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260811 22:31:10.778394 7700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:10.777952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:39473 (local address 127.0.0.1:55018)
0811 22:31:10.778167 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:10.778259 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:39473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260811 22:31:10.778612 7695 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.106.60:39473: Network error: Client connection negotiation failed: client connection to 127.29.106.60:39473: connect: Connection refused (error 111)
I20260811 22:31:10.781437 7852 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260811 22:31:10.784765 7877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:31:10.785992 7852 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260811 22:31:10.786144 7852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/minicluster-data/master-2/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/minicluster-data/master-2/wal
uuid: "ac5daa92df8c465e83cc56ca6df19bd7"
format_stamp: "Formatted at 2026-08-11 22:31:10 on dist-test-slave-gjrz"
server_key: "6435bc985b0ddae30c71874d0bc0720d"
server_key_iv: "0ee2ad15de93cd3635df34d9e02cadae"
server_key_version: "encryptionkey@0"
I20260811 22:31:10.786329 7852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:31:10.798940 7852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:31:10.802409 7852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:31:10.802652 7852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:31:10.819080 7852 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.60:39473
I20260811 22:31:10.819100 7970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.60:39473 every 8 connection(s)
I20260811 22:31:10.820288 7852 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/minicluster-data/master-2/data/info.pb
I20260811 22:31:10.823860 7972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:31:10.827934 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 7852
I20260811 22:31:10.828126 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786487105757681-30120-0/minicluster-data/master-2/wal/instance
I20260811 22:31:10.828719 30120 external_mini_cluster.cc:1428] Setting key 4e1f96b27127f0c9265bad6721ea5827
I20260811 22:31:10.830677 7972 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.106.62" port: 36013 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 22:31:10 dist-test-slave-gjrz krb5kdc[7519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487470, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.60@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
Aug 11 22:31:10 dist-test-slave-gjrz krb5kdc[7519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.60@KRBTEST.COM
I20260811 22:31:10.855908 7704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:10.834409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38236 (local address 127.29.106.62:36013)
0811 22:31:10.834656 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:10.834663 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:10.839650 (+ 4987us) server_negotiation.cc:407] Connection header received
0811 22:31:10.841506 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:10.841513 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:10.841588 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:10.841707 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:10.843186 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:10.844220 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:10.845949 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:10.846379 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:10.846502 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:10.850346 (+ 3844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:31:10.850352 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:31:10.850357 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:31:10.850396 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:31:10.852969 (+ 2573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:10.853722 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:10.853727 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:10.853729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:10.853858 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:10.854267 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:10.854270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:10.854272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:10.854520 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:31:10.854658 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:10.855351 (+ 693us) server_negotiation.cc:299] Negotiation successful
0811 22:31:10.855571 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260811 22:31:10.856868 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:10.833906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:36013 (local address 127.0.0.1:38236)
0811 22:31:10.839478 (+ 5572us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:10.839534 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0811 22:31:10.841276 (+ 1742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:31:10.841794 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:31:10.841800 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:31:10.842545 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:31:10.843010 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:31:10.843035 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:31:10.844387 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:31:10.844391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:31:10.845803 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:31:10.845820 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:31:10.846605 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:31:10.846610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:31:10.846811 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:10.847596 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:31:10.847625 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:31:10.850149 (+ 2524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:31:10.853105 (+ 2956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:31:10.853118 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:31:10.853133 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:31:10.853606 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:31:10.853969 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:31:10.853974 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:31:10.853977 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:31:10.854155 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:31:10.854657 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:31:10.854664 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:31:10.854978 (+ 314us) client_negotiation.cc:769] Sending connection context
0811 22:31:10.855203 (+ 225us) client_negotiation.cc:240] Negotiation successful
0811 22:31:10.855652 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5309,"thread_start_us":5188,"threads_started":1}
I20260811 22:31:10.856889 7982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:10.830421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55024 (local address 127.29.106.60:39473)
0811 22:31:10.839055 (+ 8634us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:10.839069 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:10.839721 (+ 652us) server_negotiation.cc:407] Connection header received
0811 22:31:10.841282 (+ 1561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:10.841306 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:10.841546 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:10.841753 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:10.843499 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:10.844947 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:10.846437 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:10.847065 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:10.847189 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:10.851428 (+ 4239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:31:10.851436 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:31:10.851447 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:31:10.851475 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:31:10.854114 (+ 2639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:10.854773 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:10.854780 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:10.854782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:10.854968 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:10.855363 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:10.855367 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:10.855369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:10.855676 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:31:10.855992 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:10.856249 (+ 257us) server_negotiation.cc:299] Negotiation successful
0811 22:31:10.856473 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8406,"spinlock_wait_cycles":20096,"thread_start_us":8198,"threads_started":1}
I20260811 22:31:10.858384 7972 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.106.61" port: 40697 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 22:31:10 dist-test-slave-gjrz krb5kdc[7519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487470, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.60@KRBTEST.COM for kudu/127.29.106.61@KRBTEST.COM
Aug 11 22:31:10 dist-test-slave-gjrz krb5kdc[7519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:31:10.870867 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:10.859139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50414 (local address 127.29.106.61:40697)
0811 22:31:10.859352 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:10.859358 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:10.859391 (+ 33us) server_negotiation.cc:407] Connection header received
0811 22:31:10.859550 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:10.859555 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:10.859623 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:10.859702 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:10.860497 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:10.861229 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:10.862369 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:10.862784 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:10.862890 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:10.865831 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:31:10.865836 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:31:10.865839 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:31:10.865871 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:31:10.868481 (+ 2610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:10.869147 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:10.869151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:10.869153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:10.869294 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:10.869698 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:10.869701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:10.869703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:10.869949 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:31:10.870074 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:10.870448 (+ 374us) server_negotiation.cc:299] Negotiation successful
0811 22:31:10.870611 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260811 22:31:10.870981 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:10.859162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.61:40697 (local address 127.0.0.1:50414)
0811 22:31:10.859320 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:10.859338 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 22:31:10.859423 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:31:10.859742 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:31:10.859747 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:31:10.860158 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:31:10.860321 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:31:10.860332 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:31:10.861377 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:31:10.861381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:31:10.862229 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:31:10.862239 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:31:10.862918 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:31:10.862924 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:31:10.863092 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:10.863579 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:31:10.863595 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:31:10.865643 (+ 2048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:31:10.868602 (+ 2959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:31:10.868605 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:31:10.868608 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:31:10.869029 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:31:10.869411 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:31:10.869415 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:31:10.869419 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:31:10.869589 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:31:10.870081 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:31:10.870086 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:31:10.870242 (+ 156us) client_negotiation.cc:769] Sending connection context
0811 22:31:10.870390 (+ 148us) client_negotiation.cc:240] Negotiation successful
0811 22:31:10.870641 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260811 22:31:10.871918 7972 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.106.60" port: 39473 } has no permanent_uuid. Determining permanent_uuid...
I20260811 22:31:10.873744 7704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:10.859899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38244 (local address 127.29.106.62:36013)
0811 22:31:10.860038 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:10.860042 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:10.860070 (+ 28us) server_negotiation.cc:407] Connection header received
0811 22:31:10.860268 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:10.860272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:10.860326 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:10.860438 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:10.861602 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:10.862343 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:10.863600 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:10.863980 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:10.864083 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:10.868095 (+ 4012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:31:10.868100 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:31:10.868103 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:31:10.868129 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:31:10.870987 (+ 2858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:10.871718 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:10.871722 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:10.871724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:10.871861 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:10.872236 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:10.872241 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:10.872244 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:10.872593 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:31:10.872856 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:10.873351 (+ 495us) server_negotiation.cc:299] Negotiation successful
0811 22:31:10.873519 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Aug 11 22:31:10 dist-test-slave-gjrz krb5kdc[7519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487470, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.60@KRBTEST.COM for kudu/127.29.106.60@KRBTEST.COM
Aug 11 22:31:10 dist-test-slave-gjrz krb5kdc[7519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.61@KRBTEST.COM
I20260811 22:31:10.884404 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:10.872590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:39473 (local address 127.0.0.1:55036)
0811 22:31:10.872752 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:10.872769 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 22:31:10.872895 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:31:10.873175 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:31:10.873180 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:31:10.873629 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:31:10.873789 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:31:10.873800 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:31:10.874705 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:31:10.874710 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:31:10.875911 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:31:10.875925 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:31:10.876577 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:31:10.876583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:31:10.876764 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:10.877229 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:31:10.877251 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:31:10.879425 (+ 2174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:31:10.882262 (+ 2837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:31:10.882265 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:31:10.882268 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:31:10.882627 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:31:10.882991 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:31:10.882995 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:31:10.882998 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:31:10.883169 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:31:10.883683 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:31:10.883686 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:31:10.883809 (+ 123us) client_negotiation.cc:769] Sending connection context
0811 22:31:10.883922 (+ 113us) client_negotiation.cc:240] Negotiation successful
0811 22:31:10.884091 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260811 22:31:10.884595 7982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:10.872641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55036 (local address 127.29.106.60:39473)
0811 22:31:10.872801 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:10.872805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:10.872820 (+ 15us) server_negotiation.cc:407] Connection header received
0811 22:31:10.872998 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:10.873001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:10.873051 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:10.873119 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:10.873945 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:10.874585 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:10.876041 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:10.876438 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:10.876542 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:10.879602 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:31:10.879608 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:31:10.879613 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:31:10.879650 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:31:10.882125 (+ 2475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:10.882738 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:10.882742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:10.882744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:10.882855 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:10.883305 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:10.883310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:10.883314 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:10.883549 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:31:10.883701 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:10.884055 (+ 354us) server_negotiation.cc:299] Negotiation successful
0811 22:31:10.884256 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260811 22:31:10.889495 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:10.875328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50424 (local address 127.29.106.61:40697)
0811 22:31:10.875485 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:10.875490 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:10.875543 (+ 53us) server_negotiation.cc:407] Connection header received
0811 22:31:10.875649 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:10.875653 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:10.875717 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:10.875811 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:10.876852 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:10.877862 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:10.879074 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:10.879559 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:10.879679 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:10.883665 (+ 3986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:31:10.883671 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:31:10.883675 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:31:10.883709 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:31:10.886911 (+ 3202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:10.887556 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:10.887561 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:10.887565 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:10.887721 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:10.888088 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:10.888093 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:10.888096 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:10.888407 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:31:10.888652 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:10.888951 (+ 299us) server_negotiation.cc:299] Negotiation successful
0811 22:31:10.889174 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260811 22:31:10.890316 7972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac5daa92df8c465e83cc56ca6df19bd7: Bootstrap starting.
I20260811 22:31:10.893959 7972 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac5daa92df8c465e83cc56ca6df19bd7: Neither blocks nor log segments found. Creating new log.
I20260811 22:31:10.894198 30120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:31:10.895078 7972 log.cc:824] T 00000000000000000000000000000000 P ac5daa92df8c465e83cc56ca6df19bd7: Log is configured to *not* fsync() on all Append() calls
I20260811 22:31:10.898115 7972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac5daa92df8c465e83cc56ca6df19bd7: No bootstrap required, opened a new log
I20260811 22:31:10.897195 7647 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "60ff5f9a5fe246eaae40121a81ca46f8"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38244
I20260811 22:31:10.906781 7972 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac5daa92df8c465e83cc56ca6df19bd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ff5f9a5fe246eaae40121a81ca46f8" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 36013 } } peers { permanent_uuid: "824f0c26bfad41fcaac01dd3ba274b49" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 40697 } } peers { permanent_uuid: "ac5daa92df8c465e83cc56ca6df19bd7" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 39473 } }
I20260811 22:31:10.907069 7972 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac5daa92df8c465e83cc56ca6df19bd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac5daa92df8c465e83cc56ca6df19bd7, State: Initialized, Role: FOLLOWER
I20260811 22:31:10.907542 7972 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac5daa92df8c465e83cc56ca6df19bd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ff5f9a5fe246eaae40121a81ca46f8" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 36013 } } peers { permanent_uuid: "824f0c26bfad41fcaac01dd3ba274b49" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 40697 } } peers { permanent_uuid: "ac5daa92df8c465e83cc56ca6df19bd7" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 39473 } }
I20260811 22:31:10.908485 7985 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac5daa92df8c465e83cc56ca6df19bd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ff5f9a5fe246eaae40121a81ca46f8" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 36013 } } peers { permanent_uuid: "824f0c26bfad41fcaac01dd3ba274b49" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 40697 } } peers { permanent_uuid: "ac5daa92df8c465e83cc56ca6df19bd7" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 39473 } } }
I20260811 22:31:10.908797 7985 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac5daa92df8c465e83cc56ca6df19bd7 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:31:10.909664 7972 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac5daa92df8c465e83cc56ca6df19bd7 [sys.catalog]: configured and running, proceeding with master startup.
W20260811 22:31:10.918709 8013 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ac5daa92df8c465e83cc56ca6df19bd7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 22:31:10.918831 8013 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 22:31:10.916582 7852 master_runner.cc:186] Error getting master registration for 127.29.106.62:36013: OK, instance_id { permanent_uuid: "60ff5f9a5fe246eaae40121a81ca46f8" instance_seqno: 1786487469822970 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260811 22:31:10.927327 7852 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260811 22:31:10.955816 7695 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.106.60" port: 39473 } attempt: 4
Aug 11 22:31:10 dist-test-slave-gjrz krb5kdc[7519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487469, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for kudu/127.29.106.60@KRBTEST.COM
I20260811 22:31:10.971808 7700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:10.956465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:39473 (local address 127.0.0.1:55052)
0811 22:31:10.956664 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:10.956685 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 22:31:10.956796 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:31:10.957202 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:31:10.957208 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:31:10.957715 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:31:10.957898 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:31:10.957909 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:31:10.959341 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:31:10.959347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:31:10.960841 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:31:10.960858 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:31:10.961853 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:31:10.961859 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:31:10.962067 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:10.962714 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:31:10.962731 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:31:10.965125 (+ 2394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:31:10.969017 (+ 3892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:31:10.969023 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:31:10.969026 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:31:10.969524 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:31:10.969914 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:31:10.969919 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:31:10.969923 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:31:10.970087 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:31:10.970789 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:31:10.970794 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:31:10.970969 (+ 175us) client_negotiation.cc:769] Sending connection context
0811 22:31:10.971119 (+ 150us) client_negotiation.cc:240] Negotiation successful
0811 22:31:10.971404 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260811 22:31:10.971869 7982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:10.956572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55052 (local address 127.29.106.60:39473)
0811 22:31:10.956798 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:10.956804 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:10.956827 (+ 23us) server_negotiation.cc:407] Connection header received
0811 22:31:10.956904 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:10.956909 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:10.956991 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:10.957130 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:10.958046 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:10.959163 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:10.961013 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:10.961702 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:10.961838 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:10.965317 (+ 3479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:31:10.965324 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:31:10.965329 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:31:10.965366 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:31:10.968877 (+ 3511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:10.969646 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:10.969652 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:10.969655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:10.969802 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:10.970194 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:10.970199 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:10.970202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:10.970674 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:31:10.970846 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:10.971231 (+ 385us) server_negotiation.cc:299] Negotiation successful
0811 22:31:10.971483 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260811 22:31:10.977634 7695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60ff5f9a5fe246eaae40121a81ca46f8: Bootstrap starting.
I20260811 22:31:10.980839 7695 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60ff5f9a5fe246eaae40121a81ca46f8: Neither blocks nor log segments found. Creating new log.
I20260811 22:31:10.981846 7695 log.cc:824] T 00000000000000000000000000000000 P 60ff5f9a5fe246eaae40121a81ca46f8: Log is configured to *not* fsync() on all Append() calls
I20260811 22:31:10.984531 7695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60ff5f9a5fe246eaae40121a81ca46f8: No bootstrap required, opened a new log
I20260811 22:31:10.989652 7695 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60ff5f9a5fe246eaae40121a81ca46f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ff5f9a5fe246eaae40121a81ca46f8" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 36013 } } peers { permanent_uuid: "824f0c26bfad41fcaac01dd3ba274b49" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 40697 } } peers { permanent_uuid: "ac5daa92df8c465e83cc56ca6df19bd7" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 39473 } }
I20260811 22:31:10.990070 7695 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60ff5f9a5fe246eaae40121a81ca46f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ff5f9a5fe246eaae40121a81ca46f8, State: Initialized, Role: FOLLOWER
I20260811 22:31:10.990513 7695 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60ff5f9a5fe246eaae40121a81ca46f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ff5f9a5fe246eaae40121a81ca46f8" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 36013 } } peers { permanent_uuid: "824f0c26bfad41fcaac01dd3ba274b49" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 40697 } } peers { permanent_uuid: "ac5daa92df8c465e83cc56ca6df19bd7" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 39473 } }
I20260811 22:31:10.991458 8020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60ff5f9a5fe246eaae40121a81ca46f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ff5f9a5fe246eaae40121a81ca46f8" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 36013 } } peers { permanent_uuid: "824f0c26bfad41fcaac01dd3ba274b49" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 40697 } } peers { permanent_uuid: "ac5daa92df8c465e83cc56ca6df19bd7" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 39473 } } }
I20260811 22:31:10.991858 8020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60ff5f9a5fe246eaae40121a81ca46f8 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:31:10.992604 7695 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60ff5f9a5fe246eaae40121a81ca46f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:31:10.992792 7647 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60ff5f9a5fe246eaae40121a81ca46f8 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260811 22:31:10.993021 7647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60ff5f9a5fe246eaae40121a81ca46f8 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:31:10.996637 7647 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60ff5f9a5fe246eaae40121a81ca46f8 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ff5f9a5fe246eaae40121a81ca46f8" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 36013 } } peers { permanent_uuid: "824f0c26bfad41fcaac01dd3ba274b49" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 40697 } } peers { permanent_uuid: "ac5daa92df8c465e83cc56ca6df19bd7" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 39473 } }
I20260811 22:31:10.999228 7922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "60ff5f9a5fe246eaae40121a81ca46f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ac5daa92df8c465e83cc56ca6df19bd7"
I20260811 22:31:10.999799 7922 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac5daa92df8c465e83cc56ca6df19bd7 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:31:11.003654 7922 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ac5daa92df8c465e83cc56ca6df19bd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ff5f9a5fe246eaae40121a81ca46f8 in term 1.
I20260811 22:31:11.004434 7596 leader_election.cc:302] T 00000000000000000000000000000000 P 60ff5f9a5fe246eaae40121a81ca46f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ff5f9a5fe246eaae40121a81ca46f8, ac5daa92df8c465e83cc56ca6df19bd7; no voters:
I20260811 22:31:11.004908 8020 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60ff5f9a5fe246eaae40121a81ca46f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:31:11.005160 7647 leader_election.cc:288] T 00000000000000000000000000000000 P 60ff5f9a5fe246eaae40121a81ca46f8 [CANDIDATE]: Term 1 election: Requested vote from peers 824f0c26bfad41fcaac01dd3ba274b49 (127.29.106.61:40697), ac5daa92df8c465e83cc56ca6df19bd7 (127.29.106.60:39473)
I20260811 22:31:11.023913 8020 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60ff5f9a5fe246eaae40121a81ca46f8 [term 1 LEADER]: Becoming Leader. State: Replica: 60ff5f9a5fe246eaae40121a81ca46f8, State: Running, Role: LEADER
I20260811 22:31:11.024628 8020 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60ff5f9a5fe246eaae40121a81ca46f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ff5f9a5fe246eaae40121a81ca46f8" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 36013 } } peers { permanent_uuid: "824f0c26bfad41fcaac01dd3ba274b49" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 40697 } } peers { permanent_uuid: "ac5daa92df8c465e83cc56ca6df19bd7" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 39473 } }
I20260811 22:31:11.032445 8029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60ff5f9a5fe246eaae40121a81ca46f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60ff5f9a5fe246eaae40121a81ca46f8. Latest consensus state: current_term: 1 leader_uuid: "60ff5f9a5fe246eaae40121a81ca46f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ff5f9a5fe246eaae40121a81ca46f8" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 36013 } } peers { permanent_uuid: "824f0c26bfad41fcaac01dd3ba274b49" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 40697 } } peers { permanent_uuid: "ac5daa92df8c465e83cc56ca6df19bd7" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 39473 } } }
I20260811 22:31:11.032902 8029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60ff5f9a5fe246eaae40121a81ca46f8 [sys.catalog]: This master's current role is: LEADER
I20260811 22:31:11.035595 8045 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:31:11.044327 7837 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.106.60" port: 39473 } attempt: 5
I20260811 22:31:11.047819 8045 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:31:11.054037 7982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:11.018953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55066 (local address 127.29.106.60:39473)
0811 22:31:11.019127 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:11.019133 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:11.029468 (+ 10335us) server_negotiation.cc:407] Connection header received
0811 22:31:11.029840 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:11.029847 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:11.029932 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:11.030004 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:11.033414 (+ 3410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:11.034578 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:11.037856 (+ 3278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:11.038573 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:11.038672 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:11.044126 (+ 5454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:31:11.044133 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:31:11.044139 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:31:11.044180 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:31:11.048606 (+ 4426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:11.049938 (+ 1332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:11.049945 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:11.049948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:11.050264 (+ 316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:11.051016 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:11.051050 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:11.051053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:11.051673 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:31:11.052120 (+ 447us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:11.053011 (+ 891us) server_negotiation.cc:299] Negotiation successful
0811 22:31:11.053403 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260811 22:31:11.054495 7889 tablet.cc:2369] T 00000000000000000000000000000000 P ac5daa92df8c465e83cc56ca6df19bd7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 22:31:11.062973 7605 tablet.cc:2369] T 00000000000000000000000000000000 P 60ff5f9a5fe246eaae40121a81ca46f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 11 22:31:11 dist-test-slave-gjrz krb5kdc[7519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487470, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.61@KRBTEST.COM for kudu/127.29.106.60@KRBTEST.COM
I20260811 22:31:11.067767 7922 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ac5daa92df8c465e83cc56ca6df19bd7 [term 1 FOLLOWER]: Refusing update from remote peer 60ff5f9a5fe246eaae40121a81ca46f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:31:11.068817 8029 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 60ff5f9a5fe246eaae40121a81ca46f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac5daa92df8c465e83cc56ca6df19bd7" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 39473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:31:11.076848 7704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:11.017475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38248 (local address 127.29.106.62:36013)
0811 22:31:11.017663 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:11.017669 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:11.021119 (+ 3450us) server_negotiation.cc:407] Connection header received
0811 22:31:11.022168 (+ 1049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:11.022175 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:11.022264 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:11.022340 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:11.034514 (+ 12174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:11.035629 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:11.040389 (+ 4760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:11.041134 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:11.041235 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:11.066491 (+ 25256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:31:11.066498 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:31:11.066502 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:31:11.066566 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:31:11.070259 (+ 3693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:11.073897 (+ 3638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:11.073904 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:11.073908 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:11.074108 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:11.074577 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:11.074582 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:11.074586 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:11.075110 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:31:11.075436 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:11.076267 (+ 831us) server_negotiation.cc:299] Negotiation successful
0811 22:31:11.076500 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260811 22:31:11.080029 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:11.044858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:39473 (local address 127.0.0.1:55074)
0811 22:31:11.045221 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:11.045238 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 22:31:11.045336 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:31:11.056005 (+ 10669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:31:11.056011 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:31:11.056489 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:31:11.056719 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:31:11.056732 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:31:11.059034 (+ 2302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:31:11.059040 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:31:11.060577 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:31:11.060594 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:31:11.062260 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:31:11.062267 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:31:11.063736 (+ 1469us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:11.064321 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:31:11.064344 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:31:11.067401 (+ 3057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:31:11.073493 (+ 6092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:31:11.073498 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:31:11.073502 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:31:11.074315 (+ 813us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:31:11.075546 (+ 1231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:31:11.075586 (+ 40us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:31:11.075590 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:31:11.075870 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:31:11.077012 (+ 1142us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:31:11.077017 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:31:11.077262 (+ 245us) client_negotiation.cc:769] Sending connection context
0811 22:31:11.078816 (+ 1554us) client_negotiation.cc:240] Negotiation successful
0811 22:31:11.079337 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":302}
I20260811 22:31:11.082175 7982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:11.045538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55074 (local address 127.29.106.60:39473)
0811 22:31:11.055460 (+ 9922us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:11.055465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:11.055509 (+ 44us) server_negotiation.cc:407] Connection header received
0811 22:31:11.055621 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:11.055630 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:11.055763 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:11.056920 (+ 1157us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:11.057128 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:11.058809 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:11.060846 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:11.062007 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:11.062423 (+ 416us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:11.067744 (+ 5321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:31:11.067752 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:31:11.067757 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:31:11.067823 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:31:11.073280 (+ 5457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:11.074578 (+ 1298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:11.074584 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:11.074588 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:11.074844 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:11.076141 (+ 1297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:11.076146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:11.076150 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:11.076803 (+ 653us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:31:11.080763 (+ 3960us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:11.080981 (+ 218us) server_negotiation.cc:299] Negotiation successful
0811 22:31:11.081446 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9765,"thread_start_us":14530,"threads_started":1}
I20260811 22:31:11.087483 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:11.022485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50438 (local address 127.29.106.61:40697)
0811 22:31:11.022666 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:11.022672 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:11.034177 (+ 11505us) server_negotiation.cc:407] Connection header received
0811 22:31:11.037612 (+ 3435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:11.037619 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:11.037712 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:11.037785 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:11.042312 (+ 4527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:11.043501 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:11.061694 (+ 18193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:11.062403 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:11.062503 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:11.078459 (+ 15956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:31:11.078466 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:31:11.078471 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:31:11.078509 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:31:11.082361 (+ 3852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:11.084014 (+ 1653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:11.084021 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:11.084042 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:11.084308 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:11.085059 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:11.085064 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:11.085068 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:11.085453 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:31:11.085688 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:11.086792 (+ 1104us) server_negotiation.cc:299] Negotiation successful
0811 22:31:11.087072 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":70}
I20260811 22:31:11.088025 7837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 824f0c26bfad41fcaac01dd3ba274b49: Bootstrap starting.
I20260811 22:31:11.095086 7985 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac5daa92df8c465e83cc56ca6df19bd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60ff5f9a5fe246eaae40121a81ca46f8. Latest consensus state: current_term: 1 leader_uuid: "60ff5f9a5fe246eaae40121a81ca46f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ff5f9a5fe246eaae40121a81ca46f8" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 36013 } } peers { permanent_uuid: "824f0c26bfad41fcaac01dd3ba274b49" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 40697 } } peers { permanent_uuid: "ac5daa92df8c465e83cc56ca6df19bd7" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 39473 } } }
I20260811 22:31:11.095427 7985 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac5daa92df8c465e83cc56ca6df19bd7 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:31:11.095418 7837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 824f0c26bfad41fcaac01dd3ba274b49: Neither blocks nor log segments found. Creating new log.
I20260811 22:31:11.097332 7837 log.cc:824] T 00000000000000000000000000000000 P 824f0c26bfad41fcaac01dd3ba274b49: Log is configured to *not* fsync() on all Append() calls
I20260811 22:31:11.100891 8051 mvcc.cc:205] Tried to move back new op lower bound from 7317452681480982528 to 7317452681323499520. Current Snapshot: MvccSnapshot[applied={T|T < 7317452681480982528}]
I20260811 22:31:11.106949 7985 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac5daa92df8c465e83cc56ca6df19bd7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "60ff5f9a5fe246eaae40121a81ca46f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ff5f9a5fe246eaae40121a81ca46f8" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 36013 } } peers { permanent_uuid: "824f0c26bfad41fcaac01dd3ba274b49" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 40697 } } peers { permanent_uuid: "ac5daa92df8c465e83cc56ca6df19bd7" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 39473 } } }
I20260811 22:31:11.107194 7985 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac5daa92df8c465e83cc56ca6df19bd7 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:31:11.116206 7837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 824f0c26bfad41fcaac01dd3ba274b49: No bootstrap required, opened a new log
I20260811 22:31:11.119982 8020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60ff5f9a5fe246eaae40121a81ca46f8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "60ff5f9a5fe246eaae40121a81ca46f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ff5f9a5fe246eaae40121a81ca46f8" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 36013 } } peers { permanent_uuid: "824f0c26bfad41fcaac01dd3ba274b49" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 40697 } } peers { permanent_uuid: "ac5daa92df8c465e83cc56ca6df19bd7" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 39473 } } }
I20260811 22:31:11.120304 8020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60ff5f9a5fe246eaae40121a81ca46f8 [sys.catalog]: This master's current role is: LEADER
I20260811 22:31:11.121138 8045 catalog_manager.cc:1347] Generated new cluster ID: 6d8688baa91d4f79a77e1ead7cd07221
I20260811 22:31:11.121234 8045 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:31:11.141817 7837 raft_consensus.cc:348] T 00000000000000000000000000000000 P 824f0c26bfad41fcaac01dd3ba274b49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ff5f9a5fe246eaae40121a81ca46f8" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 36013 } } peers { permanent_uuid: "824f0c26bfad41fcaac01dd3ba274b49" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 40697 } } peers { permanent_uuid: "ac5daa92df8c465e83cc56ca6df19bd7" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 39473 } }
I20260811 22:31:11.142524 7837 raft_consensus.cc:729] T 00000000000000000000000000000000 P 824f0c26bfad41fcaac01dd3ba274b49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 824f0c26bfad41fcaac01dd3ba274b49, State: Initialized, Role: FOLLOWER
I20260811 22:31:11.143352 7837 consensus_queue.cc:260] T 00000000000000000000000000000000 P 824f0c26bfad41fcaac01dd3ba274b49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ff5f9a5fe246eaae40121a81ca46f8" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 36013 } } peers { permanent_uuid: "824f0c26bfad41fcaac01dd3ba274b49" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 40697 } } peers { permanent_uuid: "ac5daa92df8c465e83cc56ca6df19bd7" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 39473 } }
I20260811 22:31:11.149624 8072 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260811 22:31:11.159000 7796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "60ff5f9a5fe246eaae40121a81ca46f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "824f0c26bfad41fcaac01dd3ba274b49"
I20260811 22:31:11.159796 7796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 824f0c26bfad41fcaac01dd3ba274b49 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:31:11.163209 7796 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 824f0c26bfad41fcaac01dd3ba274b49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ff5f9a5fe246eaae40121a81ca46f8 in term 1.
I20260811 22:31:11.163916 7795 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 824f0c26bfad41fcaac01dd3ba274b49 [term 1 FOLLOWER]: Refusing update from remote peer 60ff5f9a5fe246eaae40121a81ca46f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:31:11.154969 7554 master_runner.cc:186] Error getting master registration for 127.29.106.61:40697: OK, instance_id { permanent_uuid: "824f0c26bfad41fcaac01dd3ba274b49" instance_seqno: 1786487470321619 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260811 22:31:11.165024 7554 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260811 22:31:11.165660 8029 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 60ff5f9a5fe246eaae40121a81ca46f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "824f0c26bfad41fcaac01dd3ba274b49" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 40697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:31:11.173705 7837 sys_catalog.cc:564] T 00000000000000000000000000000000 P 824f0c26bfad41fcaac01dd3ba274b49 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:31:11.196158 8045 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:31:11.197880 8045 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:31:11.217542 8073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 824f0c26bfad41fcaac01dd3ba274b49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60ff5f9a5fe246eaae40121a81ca46f8. Latest consensus state: current_term: 1 leader_uuid: "60ff5f9a5fe246eaae40121a81ca46f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ff5f9a5fe246eaae40121a81ca46f8" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 36013 } } peers { permanent_uuid: "824f0c26bfad41fcaac01dd3ba274b49" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 40697 } } peers { permanent_uuid: "ac5daa92df8c465e83cc56ca6df19bd7" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 39473 } } }
I20260811 22:31:11.217809 8020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60ff5f9a5fe246eaae40121a81ca46f8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "60ff5f9a5fe246eaae40121a81ca46f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ff5f9a5fe246eaae40121a81ca46f8" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 36013 } } peers { permanent_uuid: "824f0c26bfad41fcaac01dd3ba274b49" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 40697 } } peers { permanent_uuid: "ac5daa92df8c465e83cc56ca6df19bd7" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 39473 } } }
I20260811 22:31:11.218053 8073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 824f0c26bfad41fcaac01dd3ba274b49 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:31:11.218104 8020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60ff5f9a5fe246eaae40121a81ca46f8 [sys.catalog]: This master's current role is: LEADER
I20260811 22:31:11.218400 8074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 824f0c26bfad41fcaac01dd3ba274b49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60ff5f9a5fe246eaae40121a81ca46f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ff5f9a5fe246eaae40121a81ca46f8" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 36013 } } peers { permanent_uuid: "824f0c26bfad41fcaac01dd3ba274b49" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 40697 } } peers { permanent_uuid: "ac5daa92df8c465e83cc56ca6df19bd7" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 39473 } } }
I20260811 22:31:11.218791 8074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 824f0c26bfad41fcaac01dd3ba274b49 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:31:11.219115 8085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 824f0c26bfad41fcaac01dd3ba274b49 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "60ff5f9a5fe246eaae40121a81ca46f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60ff5f9a5fe246eaae40121a81ca46f8" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 36013 } } peers { permanent_uuid: "824f0c26bfad41fcaac01dd3ba274b49" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 40697 } } peers { permanent_uuid: "ac5daa92df8c465e83cc56ca6df19bd7" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 39473 } } }
I20260811 22:31:11.219488 8085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 824f0c26bfad41fcaac01dd3ba274b49 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:31:11.222007 8045 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60ff5f9a5fe246eaae40121a81ca46f8: Generated new TSK 0
I20260811 22:31:11.222903 8045 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:31:11.339505 8105 catalog_manager.cc:1259] Loaded cluster ID: 6d8688baa91d4f79a77e1ead7cd07221
I20260811 22:31:11.339726 8105 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 824f0c26bfad41fcaac01dd3ba274b49: loading cluster ID for follower catalog manager: success
I20260811 22:31:11.355705 8105 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 824f0c26bfad41fcaac01dd3ba274b49: acquiring CA information for follower catalog manager: success
I20260811 22:31:11.358363 8105 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 824f0c26bfad41fcaac01dd3ba274b49: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260811 22:31:11.403165 7982 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:31:11.387385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55082 (local address 127.29.106.60:39473)
0811 22:31:11.387584 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:11.387590 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:11.401117 (+ 13527us) server_negotiation.cc:407] Connection header received
0811 22:31:11.401436 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:11.401443 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:11.401544 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:11.401643 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:11.402926 (+ 1283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55082: BlockingRecv error: recv got EOF from 127.0.0.1:55082 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260811 22:31:11.405354 7847 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:31:11.387040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50444 (local address 127.29.106.61:40697)
0811 22:31:11.387219 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:11.387226 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:11.403762 (+ 16536us) server_negotiation.cc:407] Connection header received
0811 22:31:11.404017 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:11.404023 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:11.404102 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:11.404182 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:11.405126 (+ 944us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50444: BlockingRecv error: recv got EOF from 127.0.0.1:50444 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260811 22:31:11.406605 7704 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:31:11.387189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38258 (local address 127.29.106.62:36013)
0811 22:31:11.387424 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:11.387431 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:11.404342 (+ 16911us) server_negotiation.cc:407] Connection header received
0811 22:31:11.405032 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:11.405039 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:11.405140 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:11.405221 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:11.406347 (+ 1126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38258: BlockingRecv error: recv got EOF from 127.0.0.1:38258 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
I20260811 22:31:11.472841 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:11.457933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50456 (local address 127.29.106.61:40697)
0811 22:31:11.458152 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:11.458158 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:11.464948 (+ 6790us) server_negotiation.cc:407] Connection header received
0811 22:31:11.465496 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:11.465503 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:11.465722 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:11.465853 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:31:11.466686 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:11.467811 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:11.469811 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:11.470510 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:11.470605 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:11.471380 (+ 775us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:31:11.472040 (+ 660us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:31:11.472160 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:11.472409 (+ 249us) server_negotiation.cc:299] Negotiation successful
0811 22:31:11.472509 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260811 22:31:11.513460 7982 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:31:11.508934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55086 (local address 127.29.106.60:39473)
0811 22:31:11.509148 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:11.509155 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:11.511869 (+ 2714us) server_negotiation.cc:407] Connection header received
0811 22:31:11.512108 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:11.512112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:11.512242 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:11.512338 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:11.513256 (+ 918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55086: BlockingRecv error: recv got EOF from 127.0.0.1:55086 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260811 22:31:11.523304 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:11.518013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50470 (local address 127.29.106.61:40697)
0811 22:31:11.518236 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:11.518243 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:11.518707 (+ 464us) server_negotiation.cc:407] Connection header received
0811 22:31:11.518884 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:11.518890 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:11.518962 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:11.519036 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:31:11.519509 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:11.520248 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:11.521561 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:11.522020 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:11.522131 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:11.522541 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:31:11.522669 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:31:11.522743 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:11.522897 (+ 154us) server_negotiation.cc:299] Negotiation successful
0811 22:31:11.522975 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260811 22:31:11.533206 7982 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:31:11.529614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55094 (local address 127.29.106.60:39473)
0811 22:31:11.529797 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:11.529804 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:11.531769 (+ 1965us) server_negotiation.cc:407] Connection header received
0811 22:31:11.531929 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:11.531934 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:11.532026 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:11.532113 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:11.532995 (+ 882us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55094: BlockingRecv error: recv got EOF from 127.0.0.1:55094 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 22:31:11.545463 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:11.540328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50476 (local address 127.29.106.61:40697)
0811 22:31:11.540497 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:11.540503 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:11.541066 (+ 563us) server_negotiation.cc:407] Connection header received
0811 22:31:11.541204 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:11.541210 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:11.541276 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:11.541352 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:31:11.541735 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:11.542509 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:11.543839 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:11.544258 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:11.544369 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:11.544769 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:31:11.544875 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:31:11.544948 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:11.545131 (+ 183us) server_negotiation.cc:299] Negotiation successful
0811 22:31:11.545205 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260811 22:31:11.553045 7982 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:31:11.551002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55100 (local address 127.29.106.60:39473)
0811 22:31:11.551176 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:11.551182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:11.551795 (+ 613us) server_negotiation.cc:407] Connection header received
0811 22:31:11.551957 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:11.551963 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:11.552042 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:11.552123 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:11.552807 (+ 684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55100: BlockingRecv error: recv got EOF from 127.0.0.1:55100 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260811 22:31:11.565133 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:11.559820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50486 (local address 127.29.106.61:40697)
0811 22:31:11.560048 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:11.560054 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:11.560507 (+ 453us) server_negotiation.cc:407] Connection header received
0811 22:31:11.560675 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:11.560681 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:11.560769 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:11.560840 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:31:11.561256 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:11.561998 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:11.563450 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:11.563872 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:11.563977 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:11.564429 (+ 452us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:31:11.564543 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:31:11.564593 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:11.564788 (+ 195us) server_negotiation.cc:299] Negotiation successful
0811 22:31:11.564860 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
W20260811 22:31:11.579821 7982 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:31:11.570970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55102 (local address 127.29.106.60:39473)
0811 22:31:11.571229 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:11.571234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:11.578598 (+ 7364us) server_negotiation.cc:407] Connection header received
0811 22:31:11.578766 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:11.578771 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:11.578847 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:11.578931 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:11.579655 (+ 724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55102: BlockingRecv error: recv got EOF from 127.0.0.1:55102 (error 108)
Metrics: {"server-negotiator.queue_time_us":150}
I20260811 22:31:11.597072 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:11.591373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50500 (local address 127.29.106.61:40697)
0811 22:31:11.591571 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:11.591577 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:11.592059 (+ 482us) server_negotiation.cc:407] Connection header received
0811 22:31:11.592249 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:11.592253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:11.592318 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:11.592393 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:31:11.592884 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:11.593890 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:11.595417 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:11.595825 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:11.595934 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:11.596349 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:31:11.596459 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:31:11.596535 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:11.596692 (+ 157us) server_negotiation.cc:299] Negotiation successful
0811 22:31:11.596780 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260811 22:31:11.604351 7982 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:31:11.602375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55116 (local address 127.29.106.60:39473)
0811 22:31:11.602571 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:11.602578 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:11.603065 (+ 487us) server_negotiation.cc:407] Connection header received
0811 22:31:11.603203 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:11.603209 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:11.603321 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:11.603412 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:11.604134 (+ 722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55116: BlockingRecv error: recv got EOF from 127.0.0.1:55116 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260811 22:31:11.628376 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:11.623148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50512 (local address 127.29.106.61:40697)
0811 22:31:11.623376 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:11.623382 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:11.623867 (+ 485us) server_negotiation.cc:407] Connection header received
0811 22:31:11.624048 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:11.624059 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:11.624121 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:11.624194 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:31:11.624617 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:11.625424 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:11.626724 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:11.627151 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:11.627284 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:11.627687 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:31:11.627793 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:31:11.627869 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:11.628017 (+ 148us) server_negotiation.cc:299] Negotiation successful
0811 22:31:11.628121 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260811 22:31:11.636148 7982 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:31:11.634124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55130 (local address 127.29.106.60:39473)
0811 22:31:11.634324 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:11.634330 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:11.634821 (+ 491us) server_negotiation.cc:407] Connection header received
0811 22:31:11.635003 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:11.635009 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:11.635105 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:11.635190 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:11.635928 (+ 738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55130: BlockingRecv error: recv got EOF from 127.0.0.1:55130 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260811 22:31:11.680725 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:11.675315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50526 (local address 127.29.106.61:40697)
0811 22:31:11.675578 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:11.675586 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:11.676020 (+ 434us) server_negotiation.cc:407] Connection header received
0811 22:31:11.676285 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:11.676289 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:11.676359 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:11.676436 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:31:11.676861 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:11.677627 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:11.678956 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:11.679494 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:11.679615 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:11.679988 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:31:11.680113 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:31:11.680199 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:11.680354 (+ 155us) server_negotiation.cc:299] Negotiation successful
0811 22:31:11.680439 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260811 22:31:11.687945 7982 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:31:11.686191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55134 (local address 127.29.106.60:39473)
0811 22:31:11.686373 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:11.686379 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:11.686631 (+ 252us) server_negotiation.cc:407] Connection header received
0811 22:31:11.686771 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:11.686777 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:11.686869 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:11.686984 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:11.687703 (+ 719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55134: BlockingRecv error: recv got EOF from 127.0.0.1:55134 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260811 22:31:11.767000 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:11.761131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50540 (local address 127.29.106.61:40697)
0811 22:31:11.761331 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:11.761337 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:11.761681 (+ 344us) server_negotiation.cc:407] Connection header received
0811 22:31:11.761851 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:11.761858 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:11.761998 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:11.762122 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:31:11.762487 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:11.763720 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:11.765109 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:11.765748 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:11.765913 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:11.766264 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:31:11.766401 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:31:11.766498 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:11.766605 (+ 107us) server_negotiation.cc:299] Negotiation successful
0811 22:31:11.766690 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260811 22:31:11.777171 7982 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:31:11.775325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55142 (local address 127.29.106.60:39473)
0811 22:31:11.775511 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:11.775517 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:11.775778 (+ 261us) server_negotiation.cc:407] Connection header received
0811 22:31:11.775975 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:11.775980 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:11.776086 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:11.776166 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:11.776950 (+ 784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55142: BlockingRecv error: recv got EOF from 127.0.0.1:55142 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260811 22:31:11.913621 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:11.908698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50546 (local address 127.29.106.61:40697)
0811 22:31:11.908859 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:11.908864 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:11.909197 (+ 333us) server_negotiation.cc:407] Connection header received
0811 22:31:11.909360 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:11.909364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:11.909434 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:11.909517 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:31:11.909887 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:11.910661 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:11.912031 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:11.912580 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:11.912698 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:11.913049 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:31:11.913165 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:31:11.913231 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:11.913337 (+ 106us) server_negotiation.cc:299] Negotiation successful
0811 22:31:11.913401 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260811 22:31:11.920711 7982 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:31:11.918841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55156 (local address 127.29.106.60:39473)
0811 22:31:11.919087 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:11.919093 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:11.919293 (+ 200us) server_negotiation.cc:407] Connection header received
0811 22:31:11.919484 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:11.919488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:11.919565 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:11.919647 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:11.920472 (+ 825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55156: BlockingRecv error: recv got EOF from 127.0.0.1:55156 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
I20260811 22:31:11.921283 8013 catalog_manager.cc:1259] Loaded cluster ID: 6d8688baa91d4f79a77e1ead7cd07221
I20260811 22:31:11.921370 8013 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ac5daa92df8c465e83cc56ca6df19bd7: loading cluster ID for follower catalog manager: success
I20260811 22:31:11.923921 8013 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ac5daa92df8c465e83cc56ca6df19bd7: acquiring CA information for follower catalog manager: success
I20260811 22:31:11.925625 8013 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ac5daa92df8c465e83cc56ca6df19bd7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 22:31:12.185632 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:12.180703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50558 (local address 127.29.106.61:40697)
0811 22:31:12.180907 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:12.180913 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:12.181144 (+ 231us) server_negotiation.cc:407] Connection header received
0811 22:31:12.181338 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:12.181342 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:12.181423 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:12.181499 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:31:12.181943 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:12.182677 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:12.184092 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:12.184540 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:12.184642 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:12.185048 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:31:12.185161 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:31:12.185242 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:12.185341 (+ 99us) server_negotiation.cc:299] Negotiation successful
0811 22:31:12.185407 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260811 22:31:12.196581 7982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:12.191294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55158 (local address 127.29.106.60:39473)
0811 22:31:12.191483 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:12.191489 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:12.191714 (+ 225us) server_negotiation.cc:407] Connection header received
0811 22:31:12.191843 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:12.191847 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:12.191998 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:12.192093 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:31:12.192446 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:12.193236 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:12.194624 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:12.195019 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:12.195111 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:12.195588 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:31:12.196072 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:31:12.196186 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:12.196226 (+ 40us) server_negotiation.cc:299] Negotiation successful
0811 22:31:12.196296 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260811 22:31:12.209131 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:12.202220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50560 (local address 127.29.106.61:40697)
0811 22:31:12.202385 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:12.202390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:12.203962 (+ 1572us) server_negotiation.cc:407] Connection header received
0811 22:31:12.204118 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:12.204124 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:12.204208 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:12.204288 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:31:12.204864 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:12.205770 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:12.207386 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:12.207791 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:12.207899 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:12.208427 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:31:12.208547 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:31:12.208640 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:12.208763 (+ 123us) server_negotiation.cc:299] Negotiation successful
0811 22:31:12.208842 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260811 22:31:12.210520 7982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:12.202598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55160 (local address 127.29.106.60:39473)
0811 22:31:12.202752 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:12.202758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:12.204573 (+ 1815us) server_negotiation.cc:407] Connection header received
0811 22:31:12.204842 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:12.204848 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:12.204913 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:12.204957 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:31:12.205714 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:12.206444 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:12.208548 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:12.209141 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:12.209261 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:12.209843 (+ 582us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:31:12.209964 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:31:12.210065 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:12.210211 (+ 146us) server_negotiation.cc:299] Negotiation successful
0811 22:31:12.210285 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260811 22:31:12.211715 8350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:12.202807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38262 (local address 127.29.106.62:36013)
0811 22:31:12.204699 (+ 1892us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:12.204703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:12.204718 (+ 15us) server_negotiation.cc:407] Connection header received
0811 22:31:12.204817 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:12.204822 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:12.205026 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:12.205206 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:31:12.206600 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:12.207317 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:12.209182 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:12.209842 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:12.209956 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:12.210616 (+ 660us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:31:12.211079 (+ 463us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:31:12.211162 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:12.211345 (+ 183us) server_negotiation.cc:299] Negotiation successful
0811 22:31:12.211436 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1800,"thread_start_us":1690,"threads_started":1}
I20260811 22:31:12.219873 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 7554
I20260811 22:31:12.220809 7684 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:31:12.331038 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 7554
I20260811 22:31:12.358301 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 7711
I20260811 22:31:12.359408 7830 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:31:12.464726 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 7711
I20260811 22:31:12.492216 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 7852
I20260811 22:31:12.493299 7966 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:31:12.599077 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 7852
2026-08-11T22:31:12Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4781 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:31:12 dist-test-slave-gjrz krb5kdc[8360](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:31:12 dist-test-slave-gjrz krb5kdc[8360](info): set up 2 sockets
Aug 11 22:31:12 dist-test-slave-gjrz krb5kdc[8360](info): commencing operation
krb5kdc: starting...
W20260811 22:31:13.881771 30120 mini_kdc.cc:121] Time spent starting KDC: real 1.237s user 0.001s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/krb5kdc/test-user.keytab.
Aug 11 22:31:14 dist-test-slave-gjrz krb5kdc[8360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:31:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:31:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.106.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
Entry for principal kudu/127.29.106.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab.
I20260811 22:31:14.186926 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:34229
--webserver_interface=127.29.106.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:33011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:34229,127.29.106.61:42179,127.29.106.60:34895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:31:14.521544 8383 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260811 22:31:14.521836 8383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:31:14.521948 8383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:31:14.522061 8383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:31:14.529057 8383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:31:14.529131 8383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:31:14.529201 8383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:31:14.529285 8383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:31:14.529371 8383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:31:14.538954 8383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:33011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:34229,127.29.106.61:42179,127.29.106.60:34895
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.62.keytab
--principal=kudu/127.29.106.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.62:34229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.106.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:31:14.540238 8383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:31:14.541491 8383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:31:14.553257 8392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:31:14.553197 8393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:31:14.553231 8400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:31:14.554111 8383 server_base.cc:1034] running on GCE node
I20260811 22:31:14.554795 8383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:31:14.556145 8383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:31:14.557391 8383 hybrid_clock.cc:648] HybridClock initialized: now 1786487474557329 us; error 26 us; skew 500 ppm
I20260811 22:31:14.557721 8383 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:31:14 dist-test-slave-gjrz krb5kdc[8360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487474, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:31:14.560871 8383 init.cc:377] Logged in from keytab as kudu/127.29.106.62@KRBTEST.COM (short username kudu)
I20260811 22:31:14.562281 8383 webserver.cc:460] Webserver started at http://127.29.106.62:41161/ using document root <none> and password file <none>
I20260811 22:31:14.562917 8383 fs_manager.cc:359] Metadata directory not provided
I20260811 22:31:14.563027 8383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:31:14.563364 8383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:31:14.566160 8383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/minicluster-data/master-0/data/instance:
uuid: "aea61ae98cd54d798edb577aae882692"
format_stamp: "Formatted at 2026-08-11 22:31:14 on dist-test-slave-gjrz"
server_key: "5a72586ce09c53eea37fc60cada87d21"
server_key_iv: "40edfe56723f0d78479cb08363d2dd82"
server_key_version: "encryptionkey@0"
I20260811 22:31:14.566919 8383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/minicluster-data/master-0/wal/instance:
uuid: "aea61ae98cd54d798edb577aae882692"
format_stamp: "Formatted at 2026-08-11 22:31:14 on dist-test-slave-gjrz"
server_key: "5a72586ce09c53eea37fc60cada87d21"
server_key_iv: "40edfe56723f0d78479cb08363d2dd82"
server_key_version: "encryptionkey@0"
I20260811 22:31:14.571899 8383 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260811 22:31:14.575206 8417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:31:14.576478 8383 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260811 22:31:14.576653 8383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/minicluster-data/master-0/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/minicluster-data/master-0/wal
uuid: "aea61ae98cd54d798edb577aae882692"
format_stamp: "Formatted at 2026-08-11 22:31:14 on dist-test-slave-gjrz"
server_key: "5a72586ce09c53eea37fc60cada87d21"
server_key_iv: "40edfe56723f0d78479cb08363d2dd82"
server_key_version: "encryptionkey@0"
I20260811 22:31:14.576865 8383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:31:14.594926 8383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:31:14.598855 8383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:31:14.599100 8383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:31:14.617452 8383 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.62:34229
I20260811 22:31:14.617483 8527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.62:34229 every 8 connection(s)
I20260811 22:31:14.618688 8383 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/minicluster-data/master-0/data/info.pb
I20260811 22:31:14.622211 8532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:31:14.628223 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 8383
I20260811 22:31:14.628418 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/minicluster-data/master-0/wal/instance
I20260811 22:31:14.628993 30120 external_mini_cluster.cc:1428] Setting key 70587246cab679c48955ec268782570b
I20260811 22:31:14.629144 8532 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.106.62" port: 34229 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 22:31:14 dist-test-slave-gjrz krb5kdc[8360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487474, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:31:14.651378 8536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:14.633808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35054 (local address 127.29.106.62:34229)
0811 22:31:14.634459 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:14.634470 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:14.634520 (+ 50us) server_negotiation.cc:407] Connection header received
0811 22:31:14.635792 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:14.635798 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:14.636086 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:14.636279 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:14.637389 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:14.638958 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:14.640392 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:14.641136 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:14.641312 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:14.644813 (+ 3501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:31:14.644820 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:31:14.644831 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:31:14.644858 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:31:14.647481 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:14.648093 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:14.648099 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:14.648102 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:14.648235 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:14.648608 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:14.648612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:14.648613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:14.649058 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:31:14.649237 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:14.649746 (+ 509us) server_negotiation.cc:299] Negotiation successful
0811 22:31:14.650015 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":459,"thread_start_us":316,"threads_started":1}
I20260811 22:31:14.651398 8534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:14.632568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:34229 (local address 127.0.0.1:35054)
0811 22:31:14.633347 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:14.633403 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0811 22:31:14.635442 (+ 2039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:31:14.636313 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:31:14.636318 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:31:14.636840 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:31:14.637186 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:31:14.637203 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:31:14.639135 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:31:14.639140 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:31:14.640251 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:31:14.640264 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:31:14.641348 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:31:14.641354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:31:14.641575 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:14.642207 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:31:14.642229 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:31:14.644606 (+ 2377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:31:14.647593 (+ 2987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:31:14.647604 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:31:14.647614 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:31:14.647971 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:31:14.648335 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:31:14.648339 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:31:14.648343 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:31:14.648503 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:31:14.649186 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:31:14.649193 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:31:14.649379 (+ 186us) client_negotiation.cc:769] Sending connection context
0811 22:31:14.649585 (+ 206us) client_negotiation.cc:240] Negotiation successful
0811 22:31:14.650148 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"mutex_wait_us":182,"thread_start_us":349,"threads_started":1}
I20260811 22:31:14.654743 8532 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.106.61" port: 42179 } has no permanent_uuid. Determining permanent_uuid...
I20260811 22:31:14.655653 8534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:14.655295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.61:42179 (local address 127.0.0.1:55144)
0811 22:31:14.655442 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:14.655550 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.61:42179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260811 22:31:14.655866 8423 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.106.61:42179: connect: Connection refused (error 111)
I20260811 22:31:14.657323 8534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:14.656983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.61:42179 (local address 127.0.0.1:55146)
0811 22:31:14.657136 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:14.657198 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.61:42179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260811 22:31:14.657503 8532 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.106.61:42179: Network error: Client connection negotiation failed: client connection to 127.29.106.61:42179: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.29.106.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.61.keytab.
Entry for principal kudu/127.29.106.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.61.keytab.
I20260811 22:31:14.696729 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.61:42179
--webserver_interface=127.29.106.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.61.keytab
--principal=kudu/127.29.106.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:33011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:34229,127.29.106.61:42179,127.29.106.60:34895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 22:31:14.706825 8532 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.106.61" port: 42179 } attempt: 1
I20260811 22:31:14.707986 8534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:14.707515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.61:42179 (local address 127.0.0.1:55152)
0811 22:31:14.707729 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:14.707853 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.61:42179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260811 22:31:14.708945 8534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:14.708623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.61:42179 (local address 127.0.0.1:55160)
0811 22:31:14.708738 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:14.708806 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.61:42179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260811 22:31:14.709168 8532 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.106.61:42179: Network error: Client connection negotiation failed: client connection to 127.29.106.61:42179: connect: Connection refused (error 111)
I20260811 22:31:14.777416 8532 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.106.61" port: 42179 } attempt: 2
I20260811 22:31:14.778671 8534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:14.778034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.61:42179 (local address 127.0.0.1:55172)
0811 22:31:14.778337 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:14.778464 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.61:42179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260811 22:31:14.779847 8534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:14.779375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.61:42179 (local address 127.0.0.1:55178)
0811 22:31:14.779531 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:14.779639 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.61:42179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260811 22:31:14.780077 8532 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.106.61:42179: Network error: Client connection negotiation failed: client connection to 127.29.106.61:42179: connect: Connection refused (error 111)
I20260811 22:31:14.871390 8532 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.106.61" port: 42179 } attempt: 3
I20260811 22:31:14.872735 8534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:14.872190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.61:42179 (local address 127.0.0.1:55190)
0811 22:31:14.872433 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:14.872555 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.61:42179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260811 22:31:14.873821 8534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:14.873420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.61:42179 (local address 127.0.0.1:55192)
0811 22:31:14.873563 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:14.873650 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.61:42179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260811 22:31:14.874025 8532 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.106.61:42179: Network error: Client connection negotiation failed: client connection to 127.29.106.61:42179: connect: Connection refused (error 111)
I20260811 22:31:15.017288 8532 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.106.61" port: 42179 } attempt: 4
I20260811 22:31:15.018599 8534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:15.017952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.61:42179 (local address 127.0.0.1:55200)
0811 22:31:15.018305 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:15.018419 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.61:42179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187}
I20260811 22:31:15.019676 8534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:15.019246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.61:42179 (local address 127.0.0.1:55214)
0811 22:31:15.019420 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:15.019524 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.61:42179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260811 22:31:15.019860 8532 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.106.61:42179: Network error: Client connection negotiation failed: client connection to 127.29.106.61:42179: connect: Connection refused (error 111)
W20260811 22:31:15.030023 8546 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260811 22:31:15.030298 8546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:31:15.030368 8546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:31:15.030447 8546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:31:15.037478 8546 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:31:15.037554 8546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:31:15.037609 8546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:31:15.037657 8546 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:31:15.037716 8546 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:31:15.047176 8546 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:33011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:34229,127.29.106.61:42179,127.29.106.60:34895
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.61.keytab
--principal=kudu/127.29.106.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.61:42179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.29.106.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:31:15.048293 8546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:31:15.049527 8546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:31:15.060606 8566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:31:15.060585 8561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:31:15.060680 8560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:31:15.061352 8546 server_base.cc:1034] running on GCE node
I20260811 22:31:15.062094 8546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:31:15.063412 8546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:31:15.064644 8546 hybrid_clock.cc:648] HybridClock initialized: now 1786487475064558 us; error 51 us; skew 500 ppm
I20260811 22:31:15.064942 8546 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:31:15 dist-test-slave-gjrz krb5kdc[8360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487475, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:31:15.067935 8546 init.cc:377] Logged in from keytab as kudu/127.29.106.61@KRBTEST.COM (short username kudu)
I20260811 22:31:15.069221 8546 webserver.cc:460] Webserver started at http://127.29.106.61:36275/ using document root <none> and password file <none>
I20260811 22:31:15.069849 8546 fs_manager.cc:359] Metadata directory not provided
I20260811 22:31:15.069950 8546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:31:15.070254 8546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:31:15.073129 8546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/minicluster-data/master-1/data/instance:
uuid: "0dde186cfa4e4859b7775381703bd41a"
format_stamp: "Formatted at 2026-08-11 22:31:15 on dist-test-slave-gjrz"
server_key: "69ca94d6476fa9194b77f7f7c7579272"
server_key_iv: "72c84dcf6ace36c830b487f33870f697"
server_key_version: "encryptionkey@0"
I20260811 22:31:15.073866 8546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/minicluster-data/master-1/wal/instance:
uuid: "0dde186cfa4e4859b7775381703bd41a"
format_stamp: "Formatted at 2026-08-11 22:31:15 on dist-test-slave-gjrz"
server_key: "69ca94d6476fa9194b77f7f7c7579272"
server_key_iv: "72c84dcf6ace36c830b487f33870f697"
server_key_version: "encryptionkey@0"
I20260811 22:31:15.078688 8546 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 22:31:15.082088 8581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:31:15.083310 8546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260811 22:31:15.083467 8546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/minicluster-data/master-1/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/minicluster-data/master-1/wal
uuid: "0dde186cfa4e4859b7775381703bd41a"
format_stamp: "Formatted at 2026-08-11 22:31:15 on dist-test-slave-gjrz"
server_key: "69ca94d6476fa9194b77f7f7c7579272"
server_key_iv: "72c84dcf6ace36c830b487f33870f697"
server_key_version: "encryptionkey@0"
I20260811 22:31:15.083665 8546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:31:15.101593 8546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:31:15.105363 8546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:31:15.105595 8546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:31:15.122225 8546 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.61:42179
I20260811 22:31:15.122247 8679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.61:42179 every 8 connection(s)
I20260811 22:31:15.123528 8546 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/minicluster-data/master-1/data/info.pb
I20260811 22:31:15.125159 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 8546
I20260811 22:31:15.125319 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/minicluster-data/master-1/wal/instance
I20260811 22:31:15.125763 30120 external_mini_cluster.cc:1428] Setting key 43e0befc6d458333615ddddded7db858
I20260811 22:31:15.127095 8683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:31:15.133448 8683 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.106.62" port: 34229 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 22:31:15 dist-test-slave-gjrz krb5kdc[8360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487475, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.61@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:31:15.153995 8536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:15.136017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35070 (local address 127.29.106.62:34229)
0811 22:31:15.136229 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:15.136235 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:15.137274 (+ 1039us) server_negotiation.cc:407] Connection header received
0811 22:31:15.139067 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:15.139073 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:15.139153 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:15.139286 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:15.140700 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:15.141373 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:15.143471 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:15.143888 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:15.144013 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:15.147849 (+ 3836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:31:15.147853 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:31:15.147857 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:31:15.147889 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:31:15.151039 (+ 3150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:15.151807 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:15.151813 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:15.151816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:15.151960 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:15.152320 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:15.152324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:15.152327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:15.152676 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:31:15.152828 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:15.153489 (+ 661us) server_negotiation.cc:299] Negotiation successful
0811 22:31:15.153705 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260811 22:31:15.154698 8686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:15.136340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:34229 (local address 127.0.0.1:35070)
0811 22:31:15.137101 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:15.137154 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0811 22:31:15.138813 (+ 1659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:31:15.139471 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:31:15.139478 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:31:15.139981 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:31:15.140523 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:31:15.140540 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:31:15.141542 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:31:15.141547 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:31:15.143321 (+ 1774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:31:15.143336 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:31:15.144128 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:31:15.144134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:31:15.144394 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:15.145044 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:31:15.145086 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:31:15.147630 (+ 2544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:31:15.151188 (+ 3558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:31:15.151195 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:31:15.151206 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:31:15.151668 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:31:15.152051 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:31:15.152053 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:31:15.152056 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:31:15.152211 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:31:15.152784 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:31:15.152790 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:31:15.153120 (+ 330us) client_negotiation.cc:769] Sending connection context
0811 22:31:15.153327 (+ 207us) client_negotiation.cc:240] Negotiation successful
0811 22:31:15.153668 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":314,"threads_started":1}
I20260811 22:31:15.156438 8683 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.106.61" port: 42179 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.29.106.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.106.60@KRBTEST.COM" created.
Aug 11 22:31:15 dist-test-slave-gjrz krb5kdc[8360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487475, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.61@KRBTEST.COM for kudu/127.29.106.61@KRBTEST.COM
I20260811 22:31:15.170581 8686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:15.157105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.61:42179 (local address 127.0.0.1:55226)
0811 22:31:15.157310 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:15.157331 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 22:31:15.157426 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:31:15.158278 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:31:15.158283 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:31:15.158720 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:31:15.158898 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:31:15.158911 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:31:15.160253 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:31:15.160258 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:31:15.161388 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:31:15.161401 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:31:15.162255 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:31:15.162259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:31:15.162384 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:15.162880 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:31:15.162895 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:31:15.165013 (+ 2118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:31:15.168470 (+ 3457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:31:15.168473 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:31:15.168476 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:31:15.168826 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:31:15.169185 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:31:15.169188 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:31:15.169190 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:31:15.169322 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:31:15.169924 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:31:15.169927 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:31:15.170036 (+ 109us) client_negotiation.cc:769] Sending connection context
0811 22:31:15.170158 (+ 122us) client_negotiation.cc:240] Negotiation successful
0811 22:31:15.170340 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260811 22:31:15.170893 8692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:15.157320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55226 (local address 127.29.106.61:42179)
0811 22:31:15.157882 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:15.157887 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:15.157912 (+ 25us) server_negotiation.cc:407] Connection header received
0811 22:31:15.157952 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:15.157957 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:15.158111 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:15.158216 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:15.159065 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:15.160102 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:15.161513 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:15.162130 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:15.162224 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:15.165234 (+ 3010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:31:15.165250 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:31:15.165265 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:31:15.165307 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:31:15.168354 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:15.168948 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:15.168956 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:15.168959 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:15.169096 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:15.169416 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:15.169420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:15.169422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:15.169809 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:31:15.170020 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:15.170319 (+ 299us) server_negotiation.cc:299] Negotiation successful
0811 22:31:15.170540 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":286,"threads_started":1}
I20260811 22:31:15.173169 8683 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.106.60" port: 34895 } has no permanent_uuid. Determining permanent_uuid...
I20260811 22:31:15.174156 8686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:15.173765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:34895 (local address 127.0.0.1:51438)
0811 22:31:15.173893 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:15.173999 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:34895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260811 22:31:15.174396 8585 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.106.60:34895: connect: Connection refused (error 111)
I20260811 22:31:15.175828 8686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:15.175510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:34895 (local address 127.0.0.1:51442)
0811 22:31:15.175659 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:15.175737 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:34895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260811 22:31:15.176034 8683 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.106.60:34895: Network error: Client connection negotiation failed: client connection to 127.29.106.60:34895: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.106.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.60.keytab.
Entry for principal kudu/127.29.106.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.60.keytab.
I20260811 22:31:15.200364 30120 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7tIZk9/build/asan/bin/kudu
/tmp/dist-test-task7tIZk9/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.60:34895
--webserver_interface=127.29.106.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.60.keytab
--principal=kudu/127.29.106.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.106.20:33011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.106.62:34229,127.29.106.61:42179,127.29.106.60:34895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 22:31:15.225365 8683 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.106.60" port: 34895 } attempt: 1
I20260811 22:31:15.226622 8686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:15.226083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:34895 (local address 127.0.0.1:51448)
0811 22:31:15.226312 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:15.226427 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:34895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260811 22:31:15.227747 8686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:15.227331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:34895 (local address 127.0.0.1:51452)
0811 22:31:15.227503 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:15.227605 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:34895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260811 22:31:15.227949 8683 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.106.60:34895: Network error: Client connection negotiation failed: client connection to 127.29.106.60:34895: connect: Connection refused (error 111)
I20260811 22:31:15.296180 8683 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.106.60" port: 34895 } attempt: 2
I20260811 22:31:15.297366 8686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:15.296834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:34895 (local address 127.0.0.1:51464)
0811 22:31:15.297063 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:15.297178 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:34895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260811 22:31:15.298579 8686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:15.298124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:34895 (local address 127.0.0.1:51468)
0811 22:31:15.298342 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:15.298436 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:34895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260811 22:31:15.298870 8683 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.106.60:34895: Network error: Client connection negotiation failed: client connection to 127.29.106.60:34895: connect: Connection refused (error 111)
I20260811 22:31:15.319124 8532 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.106.61" port: 42179 } attempt: 5
Aug 11 22:31:15 dist-test-slave-gjrz krb5kdc[8360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487474, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for kudu/127.29.106.61@KRBTEST.COM
I20260811 22:31:15.332091 8692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:15.320091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55228 (local address 127.29.106.61:42179)
0811 22:31:15.320277 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:15.320284 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:15.320304 (+ 20us) server_negotiation.cc:407] Connection header received
0811 22:31:15.320367 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:15.320373 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:15.320457 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:15.320541 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:15.321336 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:15.322074 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:15.323221 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:15.323695 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:15.323805 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:15.326883 (+ 3078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:31:15.326887 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:31:15.326890 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:31:15.326924 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:31:15.329561 (+ 2637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:15.330212 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:15.330216 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:15.330219 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:15.330346 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:15.330699 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:15.330702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:15.330705 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:15.330983 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:31:15.331118 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:15.331544 (+ 426us) server_negotiation.cc:299] Negotiation successful
0811 22:31:15.331769 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260811 22:31:15.332093 8534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:15.319879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.61:42179 (local address 127.0.0.1:55228)
0811 22:31:15.320092 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:15.320114 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 22:31:15.320244 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:31:15.320600 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:31:15.320605 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:31:15.321002 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:31:15.321182 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:31:15.321191 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:31:15.322208 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:31:15.322214 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:31:15.323072 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:31:15.323084 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:31:15.323834 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:31:15.323839 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:31:15.324038 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:15.324650 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:31:15.324669 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:31:15.326696 (+ 2027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:31:15.329700 (+ 3004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:31:15.329705 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:31:15.329710 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:31:15.330094 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:31:15.330455 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:31:15.330460 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:31:15.330463 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:31:15.330593 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:31:15.331103 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:31:15.331108 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:31:15.331254 (+ 146us) client_negotiation.cc:769] Sending connection context
0811 22:31:15.331427 (+ 173us) client_negotiation.cc:240] Negotiation successful
0811 22:31:15.331740 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260811 22:31:15.333209 8532 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.106.60" port: 34895 } has no permanent_uuid. Determining permanent_uuid...
I20260811 22:31:15.334117 8534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:15.333753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:34895 (local address 127.0.0.1:51472)
0811 22:31:15.333908 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:15.333980 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:34895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260811 22:31:15.334947 8534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:15.334656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:34895 (local address 127.0.0.1:51488)
0811 22:31:15.334786 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:15.334860 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:34895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260811 22:31:15.335134 8532 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.106.60:34895: Network error: Client connection negotiation failed: client connection to 127.29.106.60:34895: connect: Connection refused (error 111)
I20260811 22:31:15.386323 8532 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.106.60" port: 34895 } attempt: 1
I20260811 22:31:15.387661 8534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:15.387012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:34895 (local address 127.0.0.1:51504)
0811 22:31:15.387315 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:15.387452 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:34895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260811 22:31:15.389079 8534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:15.388615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:34895 (local address 127.0.0.1:51508)
0811 22:31:15.388820 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:15.388921 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:34895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260811 22:31:15.389441 8532 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.106.60:34895: Network error: Client connection negotiation failed: client connection to 127.29.106.60:34895: connect: Connection refused (error 111)
I20260811 22:31:15.390094 8683 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.106.60" port: 34895 } attempt: 3
I20260811 22:31:15.391230 8686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:15.390657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:34895 (local address 127.0.0.1:51520)
0811 22:31:15.390930 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:15.391039 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:34895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260811 22:31:15.392509 8686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:15.391982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:34895 (local address 127.0.0.1:51532)
0811 22:31:15.392279 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:15.392369 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:34895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206}
W20260811 22:31:15.392799 8683 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.106.60:34895: Network error: Client connection negotiation failed: client connection to 127.29.106.60:34895: connect: Connection refused (error 111)
I20260811 22:31:15.457705 8532 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.106.60" port: 34895 } attempt: 2
I20260811 22:31:15.459034 8534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:15.458443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:34895 (local address 127.0.0.1:51538)
0811 22:31:15.458710 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:15.458824 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:34895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260811 22:31:15.460189 8534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:15.459746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:34895 (local address 127.0.0.1:51554)
0811 22:31:15.459920 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:15.460027 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:34895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260811 22:31:15.460588 8532 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.106.60:34895: Network error: Client connection negotiation failed: client connection to 127.29.106.60:34895: connect: Connection refused (error 111)
I20260811 22:31:15.536015 8683 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.106.60" port: 34895 } attempt: 4
I20260811 22:31:15.537151 8686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:15.536674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:34895 (local address 127.0.0.1:51562)
0811 22:31:15.536891 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:15.536991 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:34895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260811 22:31:15.538122 8686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:15.537768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:34895 (local address 127.0.0.1:51566)
0811 22:31:15.537944 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:15.538016 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:34895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260811 22:31:15.538331 8683 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.106.60:34895: Network error: Client connection negotiation failed: client connection to 127.29.106.60:34895: connect: Connection refused (error 111)
W20260811 22:31:15.540225 8703 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260811 22:31:15.540520 8703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:31:15.540621 8703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:31:15.540732 8703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:31:15.549170 8703 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:31:15.549263 8703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:31:15.549352 8703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:31:15.549422 8703 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:31:15.549503 8703 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:31:15.558969 8703 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.106.20:33011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.29.106.62:34229,127.29.106.61:42179,127.29.106.60:34895
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/krb5kdc/kudu_127.29.106.60.keytab
--principal=kudu/127.29.106.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.106.60:34895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.29.106.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 11 Aug 2026 21:01:54 UTC on ccycloud.quasar-ltooxc.root.comops.site
build id 4581475
ASAN enabled
I20260811 22:31:15.560179 8703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:31:15.561404 8703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 22:31:15.566810 8532 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.106.60" port: 34895 } attempt: 3
I20260811 22:31:15.567905 8534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:15.567457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:34895 (local address 127.0.0.1:51578)
0811 22:31:15.567674 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:15.567755 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:34895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260811 22:31:15.568832 8534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:15.568510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:34895 (local address 127.0.0.1:51582)
0811 22:31:15.568652 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:15.568729 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.106.60:34895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260811 22:31:15.569024 8532 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.106.60:34895: Network error: Client connection negotiation failed: client connection to 127.29.106.60:34895: connect: Connection refused (error 111)
W20260811 22:31:15.573074 8716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:31:15.573148 8718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:31:15.573107 8722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:31:15.573848 8703 server_base.cc:1034] running on GCE node
I20260811 22:31:15.574512 8703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:31:15.575836 8703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:31:15.577045 8703 hybrid_clock.cc:648] HybridClock initialized: now 1786487475576989 us; error 29 us; skew 500 ppm
I20260811 22:31:15.577339 8703 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 11 22:31:15 dist-test-slave-gjrz krb5kdc[8360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487475, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:31:15.580339 8703 init.cc:377] Logged in from keytab as kudu/127.29.106.60@KRBTEST.COM (short username kudu)
I20260811 22:31:15.581835 8703 webserver.cc:460] Webserver started at http://127.29.106.60:37987/ using document root <none> and password file <none>
I20260811 22:31:15.582468 8703 fs_manager.cc:359] Metadata directory not provided
I20260811 22:31:15.582576 8703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:31:15.582863 8703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:31:15.585821 8703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/minicluster-data/master-2/data/instance:
uuid: "09fc653965b84473a3f95ca80e16ee2c"
format_stamp: "Formatted at 2026-08-11 22:31:15 on dist-test-slave-gjrz"
server_key: "99a1bcee8fd062c883e891f4a015dc81"
server_key_iv: "57742b6094a6718919955164998ffab2"
server_key_version: "encryptionkey@0"
I20260811 22:31:15.586602 8703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/minicluster-data/master-2/wal/instance:
uuid: "09fc653965b84473a3f95ca80e16ee2c"
format_stamp: "Formatted at 2026-08-11 22:31:15 on dist-test-slave-gjrz"
server_key: "99a1bcee8fd062c883e891f4a015dc81"
server_key_iv: "57742b6094a6718919955164998ffab2"
server_key_version: "encryptionkey@0"
I20260811 22:31:15.591393 8703 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.006s
I20260811 22:31:15.594512 8732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:31:15.595837 8703 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:31:15.595995 8703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/minicluster-data/master-2/data,/tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/minicluster-data/master-2/wal
uuid: "09fc653965b84473a3f95ca80e16ee2c"
format_stamp: "Formatted at 2026-08-11 22:31:15 on dist-test-slave-gjrz"
server_key: "99a1bcee8fd062c883e891f4a015dc81"
server_key_iv: "57742b6094a6718919955164998ffab2"
server_key_version: "encryptionkey@0"
I20260811 22:31:15.596181 8703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:31:15.612936 8703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:31:15.616833 8703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:31:15.617081 8703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:31:15.634133 8703 rpc_server.cc:307] RPC server started. Bound to: 127.29.106.60:34895
I20260811 22:31:15.634159 8840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.106.60:34895 every 8 connection(s)
I20260811 22:31:15.635339 8703 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/minicluster-data/master-2/data/info.pb
I20260811 22:31:15.638662 8843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:31:15.644582 30120 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7tIZk9/build/asan/bin/kudu as pid 8703
I20260811 22:31:15.644737 30120 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7tIZk9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786487105757681-30120-0/minicluster-data/master-2/wal/instance
I20260811 22:31:15.645136 30120 external_mini_cluster.cc:1428] Setting key b38b96c4a5fa48e2a9c2bbde8a3ff6ab
I20260811 22:31:15.644820 8843 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.106.62" port: 34229 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 22:31:15 dist-test-slave-gjrz krb5kdc[8360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487475, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.60@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
Aug 11 22:31:15 dist-test-slave-gjrz krb5kdc[8360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.60@KRBTEST.COM
I20260811 22:31:15.664296 8536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:15.647422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35082 (local address 127.29.106.62:34229)
0811 22:31:15.647625 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:15.647631 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:15.648409 (+ 778us) server_negotiation.cc:407] Connection header received
0811 22:31:15.649432 (+ 1023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:15.649437 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:15.649513 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:15.649621 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:15.651208 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:15.652331 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:15.654285 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:15.654855 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:15.654983 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:15.658593 (+ 3610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:31:15.658598 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:31:15.658601 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:31:15.658631 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:31:15.661266 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:15.662090 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:15.662094 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:15.662096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:15.662215 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:15.662657 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:15.662661 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:15.662664 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:15.663005 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:31:15.663166 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:15.663883 (+ 717us) server_negotiation.cc:299] Negotiation successful
0811 22:31:15.664055 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260811 22:31:15.665331 8857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:15.647475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.62:34229 (local address 127.0.0.1:35082)
0811 22:31:15.648245 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:15.648284 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0811 22:31:15.649190 (+ 906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:31:15.649694 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:31:15.649701 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:31:15.650527 (+ 826us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:31:15.651031 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:31:15.651058 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:31:15.652491 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:31:15.652495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:31:15.654139 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:31:15.654156 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:31:15.655066 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:31:15.655071 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:31:15.655232 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:15.655994 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:31:15.656022 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:31:15.658410 (+ 2388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:31:15.661390 (+ 2980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:31:15.661400 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:31:15.661411 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:31:15.661976 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:31:15.662322 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:31:15.662327 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:31:15.662331 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:31:15.662562 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:31:15.663122 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:31:15.663128 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:31:15.663483 (+ 355us) client_negotiation.cc:769] Sending connection context
0811 22:31:15.663749 (+ 266us) client_negotiation.cc:240] Negotiation successful
0811 22:31:15.664141 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":353,"threads_started":1}
I20260811 22:31:15.665674 8853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:15.646664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51592 (local address 127.29.106.60:34895)
0811 22:31:15.647387 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:15.647397 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:15.647432 (+ 35us) server_negotiation.cc:407] Connection header received
0811 22:31:15.649181 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:15.649201 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:15.649458 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:15.649662 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:15.651352 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:15.652845 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:15.654514 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:15.655286 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:15.655390 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:15.659722 (+ 4332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:31:15.659730 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:31:15.659740 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:31:15.659767 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:31:15.662606 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:15.663371 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:15.663378 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:15.663381 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:15.663520 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:15.663886 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:15.663890 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:15.663894 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:15.664279 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:31:15.664587 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:15.665023 (+ 436us) server_negotiation.cc:299] Negotiation successful
0811 22:31:15.665247 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":314,"threads_started":1}
I20260811 22:31:15.666836 8843 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.106.61" port: 42179 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 22:31:15 dist-test-slave-gjrz krb5kdc[8360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487475, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.60@KRBTEST.COM for kudu/127.29.106.61@KRBTEST.COM
Aug 11 22:31:15 dist-test-slave-gjrz krb5kdc[8360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.62@KRBTEST.COM
I20260811 22:31:15.680593 8692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:15.667699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55236 (local address 127.29.106.61:42179)
0811 22:31:15.667883 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:15.667887 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:15.667902 (+ 15us) server_negotiation.cc:407] Connection header received
0811 22:31:15.667950 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:15.667954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:15.668015 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:15.668109 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:15.668960 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:15.669995 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:15.671609 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:15.672264 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:15.672390 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:15.675350 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:31:15.675356 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:31:15.675361 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:31:15.675403 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:31:15.677991 (+ 2588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:15.678724 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:15.678727 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:15.678729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:15.678858 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:15.679329 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:15.679333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:15.679335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:15.679585 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:31:15.679705 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:15.680134 (+ 429us) server_negotiation.cc:299] Negotiation successful
0811 22:31:15.680301 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260811 22:31:15.680603 8857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:15.667432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.61:42179 (local address 127.0.0.1:55236)
0811 22:31:15.667655 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:15.667671 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:31:15.667758 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:31:15.668165 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:31:15.668170 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:31:15.668634 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:31:15.668801 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:31:15.668812 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:31:15.670163 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:31:15.670168 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:31:15.671472 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:31:15.671484 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:31:15.672405 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:31:15.672409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:31:15.672541 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:15.673000 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:31:15.673022 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:31:15.675130 (+ 2108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:31:15.678125 (+ 2995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:31:15.678131 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:31:15.678134 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:31:15.678619 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:31:15.678968 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:31:15.678994 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:31:15.678996 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:31:15.679171 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:31:15.679718 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:31:15.679723 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:31:15.679871 (+ 148us) client_negotiation.cc:769] Sending connection context
0811 22:31:15.680033 (+ 162us) client_negotiation.cc:240] Negotiation successful
0811 22:31:15.680273 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135}
I20260811 22:31:15.681586 8843 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.106.60" port: 34895 } has no permanent_uuid. Determining permanent_uuid...
I20260811 22:31:15.683487 8536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:15.668329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35092 (local address 127.29.106.62:34229)
0811 22:31:15.668495 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:15.668501 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:15.668518 (+ 17us) server_negotiation.cc:407] Connection header received
0811 22:31:15.668596 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:15.668601 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:15.668666 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:15.668764 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:15.670065 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:15.671123 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:15.672700 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:15.673272 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:15.673397 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:15.677482 (+ 4085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:31:15.677486 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:31:15.677489 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:31:15.677516 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:31:15.680638 (+ 3122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:15.681284 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:15.681290 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:15.681293 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:15.681471 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:15.681863 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:15.681868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:15.681872 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:15.682153 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:31:15.682846 (+ 693us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:15.682966 (+ 120us) server_negotiation.cc:299] Negotiation successful
0811 22:31:15.683166 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Aug 11 22:31:15 dist-test-slave-gjrz krb5kdc[8360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487475, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.60@KRBTEST.COM for kudu/127.29.106.60@KRBTEST.COM
Aug 11 22:31:15 dist-test-slave-gjrz krb5kdc[8360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.106.61@KRBTEST.COM
I20260811 22:31:15.694350 8857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:15.682151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:34895 (local address 127.0.0.1:51594)
0811 22:31:15.682318 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:15.682335 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 22:31:15.682418 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:31:15.682687 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:31:15.682692 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:31:15.683151 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:31:15.683330 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:31:15.683342 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:31:15.684358 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:31:15.684363 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:31:15.685522 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:31:15.685536 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:31:15.686443 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:31:15.686447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:31:15.686623 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:15.687095 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:31:15.687110 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:31:15.689343 (+ 2233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:31:15.692222 (+ 2879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:31:15.692227 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:31:15.692231 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:31:15.692591 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:31:15.692960 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:31:15.692965 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:31:15.692968 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:31:15.693131 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:31:15.693617 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:31:15.693622 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:31:15.693759 (+ 137us) client_negotiation.cc:769] Sending connection context
0811 22:31:15.693872 (+ 113us) client_negotiation.cc:240] Negotiation successful
0811 22:31:15.694088 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260811 22:31:15.694540 8853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:15.682328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51594 (local address 127.29.106.60:34895)
0811 22:31:15.682453 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:15.682458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:15.682474 (+ 16us) server_negotiation.cc:407] Connection header received
0811 22:31:15.682512 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:15.682517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:15.682578 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:15.682649 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:15.683463 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:15.684222 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:15.685677 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:15.686303 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:15.686448 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:15.689524 (+ 3076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:31:15.689531 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:31:15.689536 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:31:15.689570 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:31:15.692086 (+ 2516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:15.692709 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:15.692715 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:15.692718 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:15.692840 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:15.693244 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:15.693247 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:15.693249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:15.693498 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:31:15.693604 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:15.693999 (+ 395us) server_negotiation.cc:299] Negotiation successful
0811 22:31:15.694212 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260811 22:31:15.698750 8692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:15.685030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55246 (local address 127.29.106.61:42179)
0811 22:31:15.685190 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:15.685195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:15.685212 (+ 17us) server_negotiation.cc:407] Connection header received
0811 22:31:15.685264 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:15.685268 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:15.685329 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:15.685423 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:15.686889 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:15.687687 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:15.689183 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:15.689678 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:15.689797 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:15.693728 (+ 3931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:31:15.693733 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:31:15.693736 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:31:15.693764 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:31:15.696444 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:15.697139 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:15.697142 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:15.697144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:15.697267 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:15.697619 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:15.697622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:15.697624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:15.697856 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:31:15.698037 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:15.698355 (+ 318us) server_negotiation.cc:299] Negotiation successful
0811 22:31:15.698516 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260811 22:31:15.700187 8843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09fc653965b84473a3f95ca80e16ee2c: Bootstrap starting.
I20260811 22:31:15.703347 30120 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:31:15.703752 8843 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09fc653965b84473a3f95ca80e16ee2c: Neither blocks nor log segments found. Creating new log.
I20260811 22:31:15.704788 8843 log.cc:824] T 00000000000000000000000000000000 P 09fc653965b84473a3f95ca80e16ee2c: Log is configured to *not* fsync() on all Append() calls
I20260811 22:31:15.708000 8843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09fc653965b84473a3f95ca80e16ee2c: No bootstrap required, opened a new log
I20260811 22:31:15.705790 8482 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "aea61ae98cd54d798edb577aae882692"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35092
I20260811 22:31:15.718550 8843 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09fc653965b84473a3f95ca80e16ee2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea61ae98cd54d798edb577aae882692" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 34229 } } peers { permanent_uuid: "0dde186cfa4e4859b7775381703bd41a" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 42179 } } peers { permanent_uuid: "09fc653965b84473a3f95ca80e16ee2c" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 34895 } }
I20260811 22:31:15.718931 8843 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09fc653965b84473a3f95ca80e16ee2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09fc653965b84473a3f95ca80e16ee2c, State: Initialized, Role: FOLLOWER
I20260811 22:31:15.719427 8843 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09fc653965b84473a3f95ca80e16ee2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea61ae98cd54d798edb577aae882692" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 34229 } } peers { permanent_uuid: "0dde186cfa4e4859b7775381703bd41a" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 42179 } } peers { permanent_uuid: "09fc653965b84473a3f95ca80e16ee2c" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 34895 } }
I20260811 22:31:15.720322 8862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09fc653965b84473a3f95ca80e16ee2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea61ae98cd54d798edb577aae882692" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 34229 } } peers { permanent_uuid: "0dde186cfa4e4859b7775381703bd41a" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 42179 } } peers { permanent_uuid: "09fc653965b84473a3f95ca80e16ee2c" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 34895 } } }
I20260811 22:31:15.720633 8862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09fc653965b84473a3f95ca80e16ee2c [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:31:15.721504 8843 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09fc653965b84473a3f95ca80e16ee2c [sys.catalog]: configured and running, proceeding with master startup.
W20260811 22:31:15.730818 8887 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 09fc653965b84473a3f95ca80e16ee2c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 22:31:15.730937 8887 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 22:31:15.728639 8703 master_runner.cc:186] Error getting master registration for 127.29.106.62:34229: OK, instance_id { permanent_uuid: "aea61ae98cd54d798edb577aae882692" instance_seqno: 1786487474612169 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260811 22:31:15.739311 8703 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260811 22:31:15.746228 8532 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.106.60" port: 34895 } attempt: 4
Aug 11 22:31:15 dist-test-slave-gjrz krb5kdc[8360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487474, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.62@KRBTEST.COM for kudu/127.29.106.60@KRBTEST.COM
I20260811 22:31:15.762504 8853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:15.747077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51610 (local address 127.29.106.60:34895)
0811 22:31:15.747344 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:15.747350 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:15.747371 (+ 21us) server_negotiation.cc:407] Connection header received
0811 22:31:15.747428 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:15.747433 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:15.747506 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:15.747618 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:15.748604 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:15.749750 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:15.750993 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:15.751797 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:15.751921 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:15.755939 (+ 4018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:31:15.755945 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:31:15.755950 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:31:15.755994 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:31:15.759526 (+ 3532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:15.760313 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:15.760318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:15.760322 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:15.760474 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:15.760893 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:15.760898 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:15.760901 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:15.761311 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:31:15.761447 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:15.761936 (+ 489us) server_negotiation.cc:299] Negotiation successful
0811 22:31:15.762162 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260811 22:31:15.762508 8534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:15.746834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:34895 (local address 127.0.0.1:51610)
0811 22:31:15.747039 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:15.747061 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 22:31:15.747187 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:31:15.747676 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:31:15.747682 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:31:15.748233 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:31:15.748444 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:31:15.748458 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:31:15.749905 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:31:15.749912 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:31:15.750845 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:31:15.750858 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:31:15.751940 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:31:15.751946 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:31:15.752177 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:15.752813 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:31:15.752836 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:31:15.755655 (+ 2819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:31:15.759671 (+ 4016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:31:15.759676 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:31:15.759680 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:31:15.760198 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:31:15.760602 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:31:15.760607 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:31:15.760611 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:31:15.760787 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:31:15.761464 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:31:15.761469 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:31:15.761650 (+ 181us) client_negotiation.cc:769] Sending connection context
0811 22:31:15.761808 (+ 158us) client_negotiation.cc:240] Negotiation successful
0811 22:31:15.762115 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260811 22:31:15.767330 8532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aea61ae98cd54d798edb577aae882692: Bootstrap starting.
I20260811 22:31:15.771281 8532 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aea61ae98cd54d798edb577aae882692: Neither blocks nor log segments found. Creating new log.
I20260811 22:31:15.772323 8532 log.cc:824] T 00000000000000000000000000000000 P aea61ae98cd54d798edb577aae882692: Log is configured to *not* fsync() on all Append() calls
I20260811 22:31:15.774906 8532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aea61ae98cd54d798edb577aae882692: No bootstrap required, opened a new log
I20260811 22:31:15.779919 8532 raft_consensus.cc:348] T 00000000000000000000000000000000 P aea61ae98cd54d798edb577aae882692 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea61ae98cd54d798edb577aae882692" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 34229 } } peers { permanent_uuid: "0dde186cfa4e4859b7775381703bd41a" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 42179 } } peers { permanent_uuid: "09fc653965b84473a3f95ca80e16ee2c" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 34895 } }
I20260811 22:31:15.780313 8532 raft_consensus.cc:729] T 00000000000000000000000000000000 P aea61ae98cd54d798edb577aae882692 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aea61ae98cd54d798edb577aae882692, State: Initialized, Role: FOLLOWER
I20260811 22:31:15.780779 8532 consensus_queue.cc:260] T 00000000000000000000000000000000 P aea61ae98cd54d798edb577aae882692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea61ae98cd54d798edb577aae882692" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 34229 } } peers { permanent_uuid: "0dde186cfa4e4859b7775381703bd41a" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 42179 } } peers { permanent_uuid: "09fc653965b84473a3f95ca80e16ee2c" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 34895 } }
I20260811 22:31:15.781770 8890 sys_catalog.cc:455] T 00000000000000000000000000000000 P aea61ae98cd54d798edb577aae882692 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea61ae98cd54d798edb577aae882692" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 34229 } } peers { permanent_uuid: "0dde186cfa4e4859b7775381703bd41a" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 42179 } } peers { permanent_uuid: "09fc653965b84473a3f95ca80e16ee2c" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 34895 } } }
I20260811 22:31:15.782198 8890 sys_catalog.cc:458] T 00000000000000000000000000000000 P aea61ae98cd54d798edb577aae882692 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:31:15.782876 8532 sys_catalog.cc:564] T 00000000000000000000000000000000 P aea61ae98cd54d798edb577aae882692 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:31:15.783015 8482 raft_consensus.cc:482] T 00000000000000000000000000000000 P aea61ae98cd54d798edb577aae882692 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260811 22:31:15.783224 8482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aea61ae98cd54d798edb577aae882692 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:31:15.786598 8482 raft_consensus.cc:504] T 00000000000000000000000000000000 P aea61ae98cd54d798edb577aae882692 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea61ae98cd54d798edb577aae882692" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 34229 } } peers { permanent_uuid: "0dde186cfa4e4859b7775381703bd41a" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 42179 } } peers { permanent_uuid: "09fc653965b84473a3f95ca80e16ee2c" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 34895 } }
I20260811 22:31:15.789157 8482 leader_election.cc:288] T 00000000000000000000000000000000 P aea61ae98cd54d798edb577aae882692 [CANDIDATE]: Term 1 election: Requested vote from peers 0dde186cfa4e4859b7775381703bd41a (127.29.106.61:42179), 09fc653965b84473a3f95ca80e16ee2c (127.29.106.60:34895)
I20260811 22:31:15.789557 8781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aea61ae98cd54d798edb577aae882692" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "09fc653965b84473a3f95ca80e16ee2c"
I20260811 22:31:15.790073 8781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09fc653965b84473a3f95ca80e16ee2c [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:31:15.793807 8781 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 09fc653965b84473a3f95ca80e16ee2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aea61ae98cd54d798edb577aae882692 in term 1.
I20260811 22:31:15.794461 8423 leader_election.cc:302] T 00000000000000000000000000000000 P aea61ae98cd54d798edb577aae882692 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09fc653965b84473a3f95ca80e16ee2c, aea61ae98cd54d798edb577aae882692; no voters:
I20260811 22:31:15.794920 8890 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aea61ae98cd54d798edb577aae882692 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:31:15.795555 8890 raft_consensus.cc:686] T 00000000000000000000000000000000 P aea61ae98cd54d798edb577aae882692 [term 1 LEADER]: Becoming Leader. State: Replica: aea61ae98cd54d798edb577aae882692, State: Running, Role: LEADER
I20260811 22:31:15.796113 8890 consensus_queue.cc:237] T 00000000000000000000000000000000 P aea61ae98cd54d798edb577aae882692 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea61ae98cd54d798edb577aae882692" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 34229 } } peers { permanent_uuid: "0dde186cfa4e4859b7775381703bd41a" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 42179 } } peers { permanent_uuid: "09fc653965b84473a3f95ca80e16ee2c" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 34895 } }
I20260811 22:31:15.799423 8904 sys_catalog.cc:455] T 00000000000000000000000000000000 P aea61ae98cd54d798edb577aae882692 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aea61ae98cd54d798edb577aae882692. Latest consensus state: current_term: 1 leader_uuid: "aea61ae98cd54d798edb577aae882692" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea61ae98cd54d798edb577aae882692" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 34229 } } peers { permanent_uuid: "0dde186cfa4e4859b7775381703bd41a" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 42179 } } peers { permanent_uuid: "09fc653965b84473a3f95ca80e16ee2c" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 34895 } } }
I20260811 22:31:15.799793 8904 sys_catalog.cc:458] T 00000000000000000000000000000000 P aea61ae98cd54d798edb577aae882692 [sys.catalog]: This master's current role is: LEADER
I20260811 22:31:15.801084 8907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:31:15.809755 8907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:31:15.827630 8928 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260811 22:31:15.837533 8683 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.106.60" port: 34895 } attempt: 5
Aug 11 22:31:15 dist-test-slave-gjrz krb5kdc[8360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487475, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.106.61@KRBTEST.COM for kudu/127.29.106.60@KRBTEST.COM
W20260811 22:31:15.851763 8433 tablet.cc:2369] T 00000000000000000000000000000000 P aea61ae98cd54d798edb577aae882692: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:31:15.861486 8536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:15.813897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35094 (local address 127.29.106.62:34229)
0811 22:31:15.814071 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:15.814077 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:15.820689 (+ 6612us) server_negotiation.cc:407] Connection header received
0811 22:31:15.820904 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:15.820910 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:15.820998 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:15.821091 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:15.833888 (+ 12797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:15.846681 (+ 12793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:15.848542 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:15.849297 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:15.849418 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:15.853207 (+ 3789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:31:15.853217 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:31:15.853223 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:31:15.853295 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:31:15.857133 (+ 3838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:15.858104 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:15.858110 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:15.858114 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:15.858296 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:15.858753 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:15.858758 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:15.858762 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:15.859137 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:31:15.859424 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:15.860838 (+ 1414us) server_negotiation.cc:299] Negotiation successful
0811 22:31:15.861085 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260811 22:31:15.865016 8686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:15.838091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.106.60:34895 (local address 127.0.0.1:51616)
0811 22:31:15.838483 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0811 22:31:15.838499 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:31:15.838608 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:31:15.839622 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:31:15.839627 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:31:15.840162 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:31:15.840347 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:31:15.840359 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:31:15.841969 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:31:15.841974 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:31:15.843298 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:31:15.843315 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:31:15.844483 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:31:15.844488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:31:15.844726 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:15.845329 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:31:15.845351 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:31:15.851699 (+ 6348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:31:15.858440 (+ 6741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:31:15.858444 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:31:15.858449 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:31:15.859289 (+ 840us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:31:15.860665 (+ 1376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:31:15.860670 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:31:15.860674 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:31:15.860931 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:31:15.863211 (+ 2280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:31:15.863216 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:31:15.863506 (+ 290us) client_negotiation.cc:769] Sending connection context
0811 22:31:15.863729 (+ 223us) client_negotiation.cc:240] Negotiation successful
0811 22:31:15.864156 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276}
I20260811 22:31:15.868315 8853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:15.838826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51616 (local address 127.29.106.60:34895)
0811 22:31:15.839294 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:15.839300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:15.839322 (+ 22us) server_negotiation.cc:407] Connection header received
0811 22:31:15.839389 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:15.839398 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:15.839482 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:15.839577 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:15.840545 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:15.841775 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:15.843475 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:15.844157 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:15.844279 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:15.852098 (+ 7819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:31:15.852105 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:31:15.852110 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:31:15.852151 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:31:15.858096 (+ 5945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:15.859659 (+ 1563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:15.859666 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:15.859669 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:15.860321 (+ 652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:15.861260 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:15.861265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:15.861268 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:15.862861 (+ 1593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:31:15.865096 (+ 2235us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:15.866043 (+ 947us) server_negotiation.cc:299] Negotiation successful
0811 22:31:15.866872 (+ 829us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320}
W20260811 22:31:15.869117 8744 tablet.cc:2369] T 00000000000000000000000000000000 P 09fc653965b84473a3f95ca80e16ee2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:31:15.874374 8683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dde186cfa4e4859b7775381703bd41a: Bootstrap starting.
I20260811 22:31:15.832062 8383 master_runner.cc:186] Error getting master registration for 127.29.106.61:42179: OK, instance_id { permanent_uuid: "0dde186cfa4e4859b7775381703bd41a" instance_seqno: 1786487475117545 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260811 22:31:15.877717 8383 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260811 22:31:15.878644 8683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0dde186cfa4e4859b7775381703bd41a: Neither blocks nor log segments found. Creating new log.
I20260811 22:31:15.886446 8683 log.cc:824] T 00000000000000000000000000000000 P 0dde186cfa4e4859b7775381703bd41a: Log is configured to *not* fsync() on all Append() calls
I20260811 22:31:15.892910 8692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:15.821414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55258 (local address 127.29.106.61:42179)
0811 22:31:15.821630 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:15.821636 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:15.839712 (+ 18076us) server_negotiation.cc:407] Connection header received
0811 22:31:15.839757 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:15.839760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:15.839826 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:15.839913 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:15.855096 (+ 15183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:15.856228 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:15.868183 (+ 11955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:15.868815 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:15.868909 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:15.883964 (+ 15055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:31:15.883971 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:31:15.883976 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:31:15.884019 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:31:15.887793 (+ 3774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:15.889175 (+ 1382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:15.889202 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:15.889205 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:15.889421 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:15.890127 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:15.890132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:15.890136 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:15.890700 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:31:15.891028 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:15.892072 (+ 1044us) server_negotiation.cc:299] Negotiation successful
0811 22:31:15.892408 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260811 22:31:15.906020 8683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dde186cfa4e4859b7775381703bd41a: No bootstrap required, opened a new log
I20260811 22:31:15.919538 8781 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 09fc653965b84473a3f95ca80e16ee2c [term 1 FOLLOWER]: Refusing update from remote peer aea61ae98cd54d798edb577aae882692: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:31:15.920642 8904 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aea61ae98cd54d798edb577aae882692 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09fc653965b84473a3f95ca80e16ee2c" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 34895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:31:15.932317 8853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:15.838990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51630 (local address 127.29.106.60:34895)
0811 22:31:15.880295 (+ 41305us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:15.880302 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:15.880423 (+ 121us) server_negotiation.cc:407] Connection header received
0811 22:31:15.880687 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:15.880694 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:15.881634 (+ 940us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:15.882527 (+ 893us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:15.883651 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:15.891520 (+ 7869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:15.894265 (+ 2745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:15.895903 (+ 1638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:15.900331 (+ 4428us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:15.902983 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:31:15.902989 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:31:15.902994 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:31:15.903028 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:31:15.913755 (+ 10727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:15.916450 (+ 2695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:15.916455 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:15.916458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:15.916620 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:31:15.917442 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:31:15.917447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:31:15.917450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:31:15.917823 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:31:15.931483 (+ 13660us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:15.931625 (+ 142us) server_negotiation.cc:299] Negotiation successful
0811 22:31:15.931914 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40669,"thread_start_us":79273,"threads_started":1}
I20260811 22:31:15.937500 8862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09fc653965b84473a3f95ca80e16ee2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader aea61ae98cd54d798edb577aae882692. Latest consensus state: current_term: 1 leader_uuid: "aea61ae98cd54d798edb577aae882692" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea61ae98cd54d798edb577aae882692" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 34229 } } peers { permanent_uuid: "0dde186cfa4e4859b7775381703bd41a" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 42179 } } peers { permanent_uuid: "09fc653965b84473a3f95ca80e16ee2c" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 34895 } } }
I20260811 22:31:15.937810 8862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09fc653965b84473a3f95ca80e16ee2c [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:31:15.939957 8904 sys_catalog.cc:455] T 00000000000000000000000000000000 P aea61ae98cd54d798edb577aae882692 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aea61ae98cd54d798edb577aae882692" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea61ae98cd54d798edb577aae882692" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 34229 } } peers { permanent_uuid: "0dde186cfa4e4859b7775381703bd41a" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 42179 } } peers { permanent_uuid: "09fc653965b84473a3f95ca80e16ee2c" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 34895 } } }
I20260811 22:31:15.940248 8904 sys_catalog.cc:458] T 00000000000000000000000000000000 P aea61ae98cd54d798edb577aae882692 [sys.catalog]: This master's current role is: LEADER
I20260811 22:31:15.942325 8938 mvcc.cc:205] Tried to move back new op lower bound from 7317452701009436672 to 7317452700865691648. Current Snapshot: MvccSnapshot[applied={T|T < 7317452701009436672}]
I20260811 22:31:15.945672 8907 catalog_manager.cc:1347] Generated new cluster ID: 2435267f9507495ca69f3705868d30ad
I20260811 22:31:15.945778 8907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:31:15.947187 8683 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0dde186cfa4e4859b7775381703bd41a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea61ae98cd54d798edb577aae882692" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 34229 } } peers { permanent_uuid: "0dde186cfa4e4859b7775381703bd41a" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 42179 } } peers { permanent_uuid: "09fc653965b84473a3f95ca80e16ee2c" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 34895 } }
I20260811 22:31:15.947599 8683 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0dde186cfa4e4859b7775381703bd41a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dde186cfa4e4859b7775381703bd41a, State: Initialized, Role: FOLLOWER
I20260811 22:31:15.948100 8683 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0dde186cfa4e4859b7775381703bd41a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea61ae98cd54d798edb577aae882692" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 34229 } } peers { permanent_uuid: "0dde186cfa4e4859b7775381703bd41a" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 42179 } } peers { permanent_uuid: "09fc653965b84473a3f95ca80e16ee2c" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 34895 } }
I20260811 22:31:15.964530 8862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09fc653965b84473a3f95ca80e16ee2c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aea61ae98cd54d798edb577aae882692" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea61ae98cd54d798edb577aae882692" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 34229 } } peers { permanent_uuid: "0dde186cfa4e4859b7775381703bd41a" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 42179 } } peers { permanent_uuid: "09fc653965b84473a3f95ca80e16ee2c" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 34895 } } }
I20260811 22:31:15.964759 8862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09fc653965b84473a3f95ca80e16ee2c [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:31:15.982932 8907 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:31:15.987337 8639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aea61ae98cd54d798edb577aae882692" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0dde186cfa4e4859b7775381703bd41a"
I20260811 22:31:15.987314 8683 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0dde186cfa4e4859b7775381703bd41a [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:31:15.987810 8637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0dde186cfa4e4859b7775381703bd41a [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:31:15.991935 8637 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0dde186cfa4e4859b7775381703bd41a [term 1 FOLLOWER]: Refusing update from remote peer aea61ae98cd54d798edb577aae882692: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:31:15.995221 8639 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0dde186cfa4e4859b7775381703bd41a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aea61ae98cd54d798edb577aae882692 in term 1.
I20260811 22:31:15.997416 8890 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aea61ae98cd54d798edb577aae882692 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dde186cfa4e4859b7775381703bd41a" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 42179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:31:16.000895 8907 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:31:15.986025 8945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dde186cfa4e4859b7775381703bd41a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea61ae98cd54d798edb577aae882692" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 34229 } } peers { permanent_uuid: "0dde186cfa4e4859b7775381703bd41a" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 42179 } } peers { permanent_uuid: "09fc653965b84473a3f95ca80e16ee2c" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 34895 } } }
I20260811 22:31:16.001320 8945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dde186cfa4e4859b7775381703bd41a [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:31:16.016324 8907 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aea61ae98cd54d798edb577aae882692: Generated new TSK 0
I20260811 22:31:16.017758 8907 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:31:16.055545 8948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dde186cfa4e4859b7775381703bd41a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aea61ae98cd54d798edb577aae882692" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea61ae98cd54d798edb577aae882692" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 34229 } } peers { permanent_uuid: "0dde186cfa4e4859b7775381703bd41a" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 42179 } } peers { permanent_uuid: "09fc653965b84473a3f95ca80e16ee2c" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 34895 } } }
I20260811 22:31:16.055954 8948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dde186cfa4e4859b7775381703bd41a [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:31:16.056351 8945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dde186cfa4e4859b7775381703bd41a [sys.catalog]: SysCatalogTable state changed. Reason: New leader aea61ae98cd54d798edb577aae882692. Latest consensus state: current_term: 1 leader_uuid: "aea61ae98cd54d798edb577aae882692" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea61ae98cd54d798edb577aae882692" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 34229 } } peers { permanent_uuid: "0dde186cfa4e4859b7775381703bd41a" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 42179 } } peers { permanent_uuid: "09fc653965b84473a3f95ca80e16ee2c" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 34895 } } }
I20260811 22:31:16.056736 8945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dde186cfa4e4859b7775381703bd41a [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:31:16.058143 8904 sys_catalog.cc:455] T 00000000000000000000000000000000 P aea61ae98cd54d798edb577aae882692 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aea61ae98cd54d798edb577aae882692" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aea61ae98cd54d798edb577aae882692" member_type: VOTER last_known_addr { host: "127.29.106.62" port: 34229 } } peers { permanent_uuid: "0dde186cfa4e4859b7775381703bd41a" member_type: VOTER last_known_addr { host: "127.29.106.61" port: 42179 } } peers { permanent_uuid: "09fc653965b84473a3f95ca80e16ee2c" member_type: VOTER last_known_addr { host: "127.29.106.60" port: 34895 } } }
I20260811 22:31:16.058432 8904 sys_catalog.cc:458] T 00000000000000000000000000000000 P aea61ae98cd54d798edb577aae882692 [sys.catalog]: This master's current role is: LEADER
I20260811 22:31:16.136053 8978 catalog_manager.cc:1259] Loaded cluster ID: 2435267f9507495ca69f3705868d30ad
I20260811 22:31:16.136175 8978 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0dde186cfa4e4859b7775381703bd41a: loading cluster ID for follower catalog manager: success
I20260811 22:31:16.139919 8978 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0dde186cfa4e4859b7775381703bd41a: acquiring CA information for follower catalog manager: success
I20260811 22:31:16.142262 8978 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0dde186cfa4e4859b7775381703bd41a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260811 22:31:16.174785 8536 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:31:16.163826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35108 (local address 127.29.106.62:34229)
0811 22:31:16.164043 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:16.164049 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:16.171462 (+ 7413us) server_negotiation.cc:407] Connection header received
0811 22:31:16.173018 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:16.173033 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:16.173128 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:16.173208 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:16.174523 (+ 1315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35108: BlockingRecv error: recv got EOF from 127.0.0.1:35108 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260811 22:31:16.178391 8692 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:31:16.172054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55266 (local address 127.29.106.61:42179)
0811 22:31:16.172233 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:16.172239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:16.176321 (+ 4082us) server_negotiation.cc:407] Connection header received
0811 22:31:16.177019 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:16.177041 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:16.177156 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:16.177228 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:16.178166 (+ 938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55266: BlockingRecv error: recv got EOF from 127.0.0.1:55266 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260811 22:31:16.182313 8853 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:31:16.179973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51638 (local address 127.29.106.60:34895)
0811 22:31:16.180154 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:16.180160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:16.180395 (+ 235us) server_negotiation.cc:407] Connection header received
0811 22:31:16.180671 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:16.180677 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:16.180757 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:16.180836 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:16.182076 (+ 1240us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51638: BlockingRecv error: recv got EOF from 127.0.0.1:51638 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260811 22:31:16.235967 8692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:16.219885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55274 (local address 127.29.106.61:42179)
0811 22:31:16.220070 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:16.220076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:16.227941 (+ 7865us) server_negotiation.cc:407] Connection header received
0811 22:31:16.228511 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:16.228518 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:16.228757 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:16.228903 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:31:16.229688 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:16.230819 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:16.232905 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:16.233586 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:16.233692 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:16.234432 (+ 740us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:31:16.235124 (+ 692us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:31:16.235241 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:16.235516 (+ 275us) server_negotiation.cc:299] Negotiation successful
0811 22:31:16.235619 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260811 22:31:16.278395 8853 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:31:16.268916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51640 (local address 127.29.106.60:34895)
0811 22:31:16.269096 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:16.269103 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:16.276368 (+ 7265us) server_negotiation.cc:407] Connection header received
0811 22:31:16.276879 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:16.276886 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:16.277016 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:16.277102 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:16.278149 (+ 1047us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51640: BlockingRecv error: recv got EOF from 127.0.0.1:51640 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260811 22:31:16.322667 8692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:16.308426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55276 (local address 127.29.106.61:42179)
0811 22:31:16.308625 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:16.308632 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:16.315886 (+ 7254us) server_negotiation.cc:407] Connection header received
0811 22:31:16.316404 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:16.316411 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:16.316501 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:16.316587 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:31:16.317292 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:16.318421 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:16.320406 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:16.321083 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:16.321180 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:16.321772 (+ 592us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:31:16.321950 (+ 178us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:31:16.322008 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:16.322269 (+ 261us) server_negotiation.cc:299] Negotiation successful
0811 22:31:16.322365 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260811 22:31:16.334048 8853 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:31:16.332046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51642 (local address 127.29.106.60:34895)
0811 22:31:16.332232 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:16.332239 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:16.332727 (+ 488us) server_negotiation.cc:407] Connection header received
0811 22:31:16.332873 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:16.332879 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:16.332957 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:16.333049 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:16.333808 (+ 759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51642: BlockingRecv error: recv got EOF from 127.0.0.1:51642 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260811 22:31:16.343923 8692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:16.338724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55288 (local address 127.29.106.61:42179)
0811 22:31:16.338907 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:16.338913 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:16.339392 (+ 479us) server_negotiation.cc:407] Connection header received
0811 22:31:16.339577 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:16.339583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:16.339658 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:16.339732 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:31:16.340124 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:16.340885 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:16.342172 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:16.342608 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:16.342715 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:16.343141 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:31:16.343301 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:31:16.343391 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:16.343500 (+ 109us) server_negotiation.cc:299] Negotiation successful
0811 22:31:16.343584 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260811 22:31:16.355003 8853 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:31:16.353081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51654 (local address 127.29.106.60:34895)
0811 22:31:16.353262 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:16.353268 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:16.353710 (+ 442us) server_negotiation.cc:407] Connection header received
0811 22:31:16.353862 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:16.353868 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:16.353944 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:16.354035 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:16.354779 (+ 744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51654: BlockingRecv error: recv got EOF from 127.0.0.1:51654 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260811 22:31:16.366847 8692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:16.361653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55296 (local address 127.29.106.61:42179)
0811 22:31:16.361901 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:16.361907 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:16.362356 (+ 449us) server_negotiation.cc:407] Connection header received
0811 22:31:16.362516 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:16.362522 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:16.362585 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:16.362657 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:31:16.363045 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:16.363824 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:16.365115 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:16.365597 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:16.365701 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:16.366129 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:31:16.366273 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:31:16.366325 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:16.366518 (+ 193us) server_negotiation.cc:299] Negotiation successful
0811 22:31:16.366598 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
W20260811 22:31:16.378700 8853 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:31:16.376743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51662 (local address 127.29.106.60:34895)
0811 22:31:16.377010 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:16.377015 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:16.377476 (+ 461us) server_negotiation.cc:407] Connection header received
0811 22:31:16.377634 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:16.377640 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:16.377723 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:16.377808 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:16.378498 (+ 690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51662: BlockingRecv error: recv got EOF from 127.0.0.1:51662 (error 108)
Metrics: {"server-negotiator.queue_time_us":154}
I20260811 22:31:16.394429 8692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:16.389427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55310 (local address 127.29.106.61:42179)
0811 22:31:16.389605 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:16.389611 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:16.390020 (+ 409us) server_negotiation.cc:407] Connection header received
0811 22:31:16.390162 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:16.390168 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:16.390233 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:16.390312 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:31:16.390709 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:16.391512 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:16.392801 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:16.393235 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:16.393337 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:16.393730 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:31:16.393854 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:31:16.393926 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:16.394077 (+ 151us) server_negotiation.cc:299] Negotiation successful
0811 22:31:16.394150 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260811 22:31:16.401582 8853 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:31:16.399842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51666 (local address 127.29.106.60:34895)
0811 22:31:16.400000 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:16.400006 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:16.400311 (+ 305us) server_negotiation.cc:407] Connection header received
0811 22:31:16.400489 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:16.400495 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:16.400563 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:16.400645 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:16.401321 (+ 676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51666: BlockingRecv error: recv got EOF from 127.0.0.1:51666 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260811 22:31:16.426066 8692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:16.420089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55316 (local address 127.29.106.61:42179)
0811 22:31:16.420278 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:16.420285 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:16.420794 (+ 509us) server_negotiation.cc:407] Connection header received
0811 22:31:16.420948 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:16.420954 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:16.421018 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:16.421100 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:31:16.421607 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:16.422461 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:16.423855 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:16.424393 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:16.424510 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:16.425096 (+ 586us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:31:16.425224 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:31:16.425321 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:16.425494 (+ 173us) server_negotiation.cc:299] Negotiation successful
0811 22:31:16.425578 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260811 22:31:16.433666 8853 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:31:16.432161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51668 (local address 127.29.106.60:34895)
0811 22:31:16.432313 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:16.432318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:16.432336 (+ 18us) server_negotiation.cc:407] Connection header received
0811 22:31:16.432533 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:16.432536 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:16.432607 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:16.432659 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:16.433507 (+ 848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51668: BlockingRecv error: recv got EOF from 127.0.0.1:51668 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260811 22:31:16.475173 8692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:16.469237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55322 (local address 127.29.106.61:42179)
0811 22:31:16.469438 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:16.469444 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:16.469762 (+ 318us) server_negotiation.cc:407] Connection header received
0811 22:31:16.469959 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:16.469966 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:16.470065 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:16.470180 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:31:16.470562 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:16.471739 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:16.473119 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:16.473847 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:16.473981 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:16.474402 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:31:16.474547 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:31:16.474629 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:16.474752 (+ 123us) server_negotiation.cc:299] Negotiation successful
0811 22:31:16.474832 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260811 22:31:16.482626 8853 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:31:16.480697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51680 (local address 127.29.106.60:34895)
0811 22:31:16.480895 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:16.480901 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:16.481201 (+ 300us) server_negotiation.cc:407] Connection header received
0811 22:31:16.481324 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:16.481329 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:16.481425 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:16.481568 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:16.482387 (+ 819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51680: BlockingRecv error: recv got EOF from 127.0.0.1:51680 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260811 22:31:16.556878 8692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:16.550227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55332 (local address 127.29.106.61:42179)
0811 22:31:16.550424 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:16.550430 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:16.551060 (+ 630us) server_negotiation.cc:407] Connection header received
0811 22:31:16.551418 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:16.551425 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:16.551524 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:16.551635 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:31:16.552260 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:16.553091 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:16.555033 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:16.555549 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:16.555653 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:16.556158 (+ 505us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:31:16.556332 (+ 174us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:31:16.556417 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:16.556576 (+ 159us) server_negotiation.cc:299] Negotiation successful
0811 22:31:16.556645 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260811 22:31:16.565920 8853 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:31:16.563874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51696 (local address 127.29.106.60:34895)
0811 22:31:16.564055 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:16.564062 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:16.564541 (+ 479us) server_negotiation.cc:407] Connection header received
0811 22:31:16.564711 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:16.564714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:16.564786 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:16.564867 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:16.565707 (+ 840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51696: BlockingRecv error: recv got EOF from 127.0.0.1:51696 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260811 22:31:16.705876 8692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:16.700909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55334 (local address 127.29.106.61:42179)
0811 22:31:16.701099 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:16.701104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:16.701461 (+ 357us) server_negotiation.cc:407] Connection header received
0811 22:31:16.701633 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:16.701638 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:16.701707 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:16.701797 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:31:16.702195 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:16.702956 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:16.704429 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:16.704830 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:16.704924 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:16.705331 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:31:16.705434 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:31:16.705496 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:16.705608 (+ 112us) server_negotiation.cc:299] Negotiation successful
0811 22:31:16.705669 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260811 22:31:16.712879 8853 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:31:16.711257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51708 (local address 127.29.106.60:34895)
0811 22:31:16.711428 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:16.711432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:16.711764 (+ 332us) server_negotiation.cc:407] Connection header received
0811 22:31:16.711896 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:16.711900 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:16.711967 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:16.712053 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:31:16.712704 (+ 651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51708: BlockingRecv error: recv got EOF from 127.0.0.1:51708 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260811 22:31:16.732868 8887 catalog_manager.cc:1259] Loaded cluster ID: 2435267f9507495ca69f3705868d30ad
I20260811 22:31:16.732959 8887 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 09fc653965b84473a3f95ca80e16ee2c: loading cluster ID for follower catalog manager: success
I20260811 22:31:16.735456 8887 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 09fc653965b84473a3f95ca80e16ee2c: acquiring CA information for follower catalog manager: success
I20260811 22:31:16.737174 8887 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 09fc653965b84473a3f95ca80e16ee2c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 22:31:16.979365 8692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:16.974571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55340 (local address 127.29.106.61:42179)
0811 22:31:16.974768 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:16.974774 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:16.975063 (+ 289us) server_negotiation.cc:407] Connection header received
0811 22:31:16.975258 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:16.975295 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:16.975384 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:16.975470 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:31:16.975951 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:16.976651 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:16.977972 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:16.978397 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:16.978487 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:16.978824 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:31:16.978927 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:31:16.978988 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:16.979090 (+ 102us) server_negotiation.cc:299] Negotiation successful
0811 22:31:16.979150 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260811 22:31:16.990165 8853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:16.984740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51714 (local address 127.29.106.60:34895)
0811 22:31:16.984919 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:16.984923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:16.985182 (+ 259us) server_negotiation.cc:407] Connection header received
0811 22:31:16.985311 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:16.985315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:16.985490 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:16.985590 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:31:16.985919 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:16.986712 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:16.988166 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:16.988634 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:16.988747 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:16.989183 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:31:16.989687 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:31:16.989789 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:16.989877 (+ 88us) server_negotiation.cc:299] Negotiation successful
0811 22:31:16.989958 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260811 22:31:17.002815 9231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:16.995485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35112 (local address 127.29.106.62:34229)
0811 22:31:16.995979 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:16.995984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:16.996004 (+ 20us) server_negotiation.cc:407] Connection header received
0811 22:31:16.996312 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:16.996318 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:16.996532 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:16.996702 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:31:16.997511 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:16.998394 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:17.000624 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:17.001095 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:17.001208 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:17.001779 (+ 571us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:31:17.002188 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:31:17.002311 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:17.002436 (+ 125us) server_negotiation.cc:299] Negotiation successful
0811 22:31:17.002526 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":253,"threads_started":1}
I20260811 22:31:17.002839 8692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:16.996072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55342 (local address 127.29.106.61:42179)
0811 22:31:16.996270 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:16.996275 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:16.996608 (+ 333us) server_negotiation.cc:407] Connection header received
0811 22:31:16.996863 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:16.996868 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:16.996962 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:16.997025 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:31:16.997901 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:16.998850 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:17.000750 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:17.001348 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:17.001453 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:17.001901 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:31:17.002036 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:31:17.002113 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:17.002433 (+ 320us) server_negotiation.cc:299] Negotiation successful
0811 22:31:17.002514 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260811 22:31:17.003670 8853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:31:16.996638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51718 (local address 127.29.106.60:34895)
0811 22:31:16.996803 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0811 22:31:16.996809 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:31:16.997406 (+ 597us) server_negotiation.cc:407] Connection header received
0811 22:31:16.997564 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:31:16.997569 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:31:16.997636 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:31:16.997708 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:31:16.998317 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:16.999144 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:17.001375 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:31:17.001985 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:31:17.002123 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:31:17.002669 (+ 546us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:31:17.002786 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:31:17.002849 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0811 22:31:17.003349 (+ 500us) server_negotiation.cc:299] Negotiation successful
0811 22:31:17.003427 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260811 22:31:17.007572 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 8383
I20260811 22:31:17.008735 8521 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:31:17.122880 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 8383
I20260811 22:31:17.159606 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 8546
I20260811 22:31:17.160987 8671 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:31:17.268121 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 8546
I20260811 22:31:17.295094 30120 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task7tIZk9/build/asan/bin/kudu pid 8703
I20260811 22:31:17.296299 8828 generic_service.cc:196] Checking for leaks (request via RPC)
I20260811 22:31:17.401942 30120 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7tIZk9/build/asan/bin/kudu with pid 8703
2026-08-11T22:31:17Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4800 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9582 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (371510 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST